LDAP performance degrades the more LDAP groups a user belongs too

Upon enabling LDAP integration for users and groups it appears the more LDAP groups a users belongs too the slower each confluence page request takes to complete. Using the performance profile feature it seems Confluence retrieves a list of all groups a user belongs to (even though only 1 or 2 of these exist in Confluence) and checks them to see if the user should have access. Can this is changed so Confluence checks if a user belongs to the defined access groups instead of the other way around and therefore the number of groups a user belongs to does not matter?

To prove this i've created two users; one belonging to just one group and the other belonging to twenty groups. The performacne profile output shows page requests for the user with twenty groups take about 60-70% longer to complete.