I have checked each user profile permission and the only option is either all contacts or only assigned contacts. It cannot be just like that. You cannot have a blanket option because there are other users that should have a degree of visibility but not all. If you can create a group for user types then give level of permissions, or customizable permission levels like email convo visibility only, user info but not email convo visibility, this would resolve some security issues