Adding new users to a space with many permissions is very cumbersome. The same issue exists for removing such users again.
Both actions require tedious and meticulous clicking. Please add a "select all" and (especially!) "deselect all" option in the permission dialogue, to make removing a user a single-click operation. Or just put an "remove user" button there, which already would help a lot.