Similar to issue CONF-1580 - our mail servers all run on non-standard ports to avoid their use by spammers. Appending the port number to the name of the mail server, e.g. my-mail.atlassian.com:26 results in the expected mail server not found error.

Using a JNDI resource is an alternative, but for ease of use it seems like this would be common enough to be useful to people.