The Bamboo CLI doesn't work unless anonymous access is enabled

Using our standard Bamboo settings I can't connect using the Bamboo CLI from a remote machine, I get the following exception:

org.swift.common.cli.CliClient$ClientException: Error - invalid request: java.rmi.RemoteException: Access is denied

When I run from a shell on the Bamboo server itself I get a different exception:

org.swift.common.cli.CliClient$ClientException: Error - invalid request: java.net.ConnectException: Connection refused

When I turn on anonymous access the command succeeds. I did some searches and found this bug:

http://jira.atlassian.com/browse/BAM-799

It was mentioned in this forum post:

http://forums.atlassian.com/thread.jspa?messageID=257241794&#257241794

Here's the relevant post:

"Do you have anonymous access turned off? If it is, do you want to try your scenarios with it turned on (i.e. non-logged in users can hit Bamboo).

I think the reason you are being redirected is because of the security filter for anonymous access. This filter should not happen with REST api calls. I have raised a bug in JIRA about this - apologies about this. http://jira.atlassian.com/browse/BAM-799"

I'm not being redirected to the login page, but allowing or disallowing anonymous users shouldn't have any effect on this call should it? The account I'm trying to connect with is in the bamboo-admin group. I've tried with a different account that is also part of the bamboo-admin group, no dice on that one either.

Am I doing something wrong, or does this just not work unless anonymous access is turned on?