Restrict access to People Directory for certain user groups

In Confluence, we have a special user group which contains only external users (i. e. users outside our company).
Users of this group should not be able to access the People Directory, because it gives them a list of all internal users (i. e. employees of our company).
We found a way to hide the "People" button in the top menu by adding a script to the Main Site Layout.
However, if the user goes to his own profile, there is another "People" link above the user name. Which leads the user back to the people directory and there's no way to hide the link.

Of course it would be even better to completely restrict access to the browsepeople.action page for certain user groups.