LDAP users which no longer exist still show up in the permissions screens as $user.fullName ($user.name).

This should show something like 'Unknown User (username)' for these users, and still allow removal or modification of the permissions.