It would be convenient if JIRA were to expose a REST API for user administration, in particular deleting users.

JIRA supports the Embedded Crowd user management API, however the authentication for the Crowd REST API relies on an application username and password, and does not accept credentials for JIRA admin users. If you have not configured application authentication, then there's no way to use this API.