So far it looks like:
If we handle the request and produce response (regular successful call) we define content-type as:
@Consumes(
{MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON})
@Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}
)
so if no content type is specified (just / in http Accept header) we use DEFAULT xml (first on the list).
This works, even in IE 
It is more complicated for exceptions. For exceptions response is build using Status class defined in atlassian-rest module.
This class does not specify any content type (however it is possible) so there is no way to enforce xml.
We can build our exceptions and build response by hand, but this will apply only for exceptions we throw.
For IE8 bug we have NotAuthenticatedException - thrown by filter if no authentication is applied. So, we don't control this exception.
This could be fixed globally in Atlassian REST, however I wonder what criteria should be used? If request specify / we could return xmk, but this should be consistent across all REST API in all products - so really should be done in Atlassian REST.
Fixed in Atlassian REST