Problem

 It is currently not possible to restrict the information from the activity stream to appropriate groups or users. It only disables it Globally. However, this cannot prevent the user to add the activity stream on the dashboard again. 

A simple example would be if the user does not have permission to access Confluence, this message will display on the Activity Stream. :

Additional information may be available, please authenticate for more information:
Authenticate with Confluence.

Clicking the X button will only remove it temporarily but once the page reloads, it will re-appear. 

Suggestion 

Introduce the ability to restrict the Activity stream based on user, group , role. etc.