Restrict access to time spent

Could it be possible to restrict access to time spent information in issues, exports, activity tab & history tab for clients ?

Today, even if we block access to worklog to certain users groups, they can access to these information by having a look at the activity or history tab, or by exporting issues, or displaying "time spent" in dashboard gadgets. It should be blocked everywhere.