The exception is one big group jira-users which has 600. It is this one that takes a long time to reload it seems. When the cache expires all of these are grabbed and this is particularly noticeable on a user picker like the reporter user picker.

Looking at the crowd logs there seems to be a separate LDAP call for each user and for each group in the whole directory. We get a bunch of entries like the following:
11:14:04,984 INFO integration.directory.connector.spring.GenericAttributesMapper: Looking for group member: baseDN = CN=A User,OU=Users,OU=MyComp,DC=mycomp,DC=nonprofit,DC=org - filter = (memberOf=CN=jira-users,OU=Groups,OU=MyComp,DC=mycomp,DC=nonprofit,DC=org)

Here is the original forum post: http://forums.atlassian.com/thread.jspa?threadID=17292&tstart=0