Click CONTACTS on the top of the page

Search for the respective person and select him/her in the middle column

On the right side of the page, click Edit

Uncheck Access Enabled

Click Save Contact to confirm your change

Read about disabling access for multiple contacts all at one time