membersOf() JQL supports teams but to use it, you must copy the teamId into the function i.e. membersOf(id: <teamId>).

In the community post, it says to find your team's ID:

1. Open the team's profile page. The URL will look like:
   https://home.atlassian.com/o/<your-org-id>/people/team/<team-id>?<site-id>

1. Copy the <team-id> part from the URL and use it in your JQL: membersOf("id:<team-id>").

You'll know it's correct when the team name appears as a lozenge in the field like so:

This isn't user-friendly - this suggestion is to have an autocomplete that lets us select the team name easily.