Permission to invite others into group

We have multiple communities of users with distinct permission groups on our site. Let's say group A and group B are disjoint sets and neither has authority over the other. I need a mechanism for group A to grant new users access to their group, but I can't give anyone in there administrator access because then they would have control over group B membership as well.

What I would like is some way for members of a group to grant other users membership in the group. It could be an attribute of the group itself.