The permissions I have set exclude certain users from posting questions, however they are still able to go directly to the URL and raise a question AND they still see the "Didn't find what you were looking for? Ask a question" panel on search results page.

STR:
1) Set group permissions on Confluence Questions.
2) Log into a user who is NOT a member of this group.
3a) Go to [base url]/questions.
3b) Perform a search.

Expected Result:
The user is not allowed to view questions or raise a question.

Actual Result:
The user is shown questions and is able to raise questions.