If a user is removed from all groups they are still being shown when viewing all users and filtering by the jira-users group

Take any user and remove that user from all groups, then go to Administration->User Browser.

Now set the group filter to Jira-Users

The user who was removed from Jira-Users is still being shown in that filtered view. That is wrong.

You should have a group filter option that lets people filter for users who are in no groups and not include them in the jira-users filtered view.