Not sure if this is HipChat or Crowd, however I had a Crowd internal directory with a bunch of users, and a group within it with only 2 users

I've setup a Hipchat application against the directory, Allow all to authenticate is set to false, and under Groups, only the group with the two users set with access.

When I sync the directory, I get 252 users synced, not just the 2 from the group I want to give permission to.

I've removed and re-added the directory and got the 252 users again