You can navigate to Admin settings > General settings > Roles.  Select the respective roles and uncheck the contact module (View, Create, Edit and Delete) and save the settings. This ensures that the users assigned to the role cannot access the contacts module. The contact module will disappear for them.