Hover over Contacts and click on My Contacts

Search for the respective person and select him/her in the middle column

On the right side of the page, click Edit

Uncheck Access Enabled

Click Save to confirm your change

Read about disabling access for multiple contacts all at one time