As stated in the article "Create and manage an Atlassian team" (under "Add or remove members of a team"), all members of your team have the same permissions, so any member of the team can invite new team members or remove other team members.

As a site-admin, it would be great if you could remove team member without having to join that team.