Issue Summary

This is reproducible on Data Center: (yes)

Bamboo 'Configure details of the mail server' fails with the following error: 

The mail settings you entered were not valid. Error thrown was: javax.mail.AuthenticationFailedException: 535 5.7.3 Authentication unsuccessful 

When attempting to configure Bamboo to connect to a Microsoft ESMTP Server, if the server has 'Anonymous access' as well as any other options such as 'Basic authentication' or 'Integrated Windows Authentication', Bamboo overlooks the 'Anonymous access' option and results in the previously mentioned error.

Steps to Reproduce

1. Configure a Microsoft ESMTP Server and add in the 'Properties' > 'Access' > 'Authentication' the check for Anonymous access
2. Configure Bamboo and run a simple email test that would result in 'Success'
3. Now add in the same 'Properties' > 'Access' > 'Authentication' another authentication method such as 'Basic Authentication' without removing 'Anonymous access'
4. Try again running a simple email test on the 'Configure details of the mail server' page and see the error message.

Expected Results

Bamboo should be able to send the email without any authentication method due to the 'Anonymous access' option being enabled (Tested and it's successfully working on Bamboo 9.2.10)

Actual Results

On Bamboo the below exception is thrown in the atlassian-bamboo.log file:

ERROR [http-nio-8085-exec-1279 url: /admin/configureMailServer.action; user: test@bamboo.com] [ConfigureMailServer] Mail server test failed: com.atlassian.mail.MailException: javax.mail.AuthenticationFailedException: 535 5.7.3 Authentication unsuccessful
    at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:227) ~[atlassian-mail-5.3.0.jar:?]    at com.atlassian.mail.server.impl.SMTPMailServerImpl.send(SMTPMailServerImpl.java:192) ~[atlassian-mail-5.3.0.jar:?]    at com.atlassian.bamboo.mail.EmailService.sendEmailSynchronously(EmailService.java:123) ~[atlassian-bamboo-core-9.5.0.jar:?]    at com.atlassian.bamboo.mail.BambooMailer.send(BambooMailer.java:70) ~[atlassian-bamboo-core-9.5.0.jar:?]    at com.atlassian.bamboo.ww2.actions.admin.mail.ConfigureMailServer.testSMTPServer(ConfigureMailServer.j 
...
...
...
Caused by: javax.mail.AuthenticationFailedException: 535 5.7.3 Authentication unsuccessful

Workaround

PS: Workaround will not work for Bamboo 10.x versions and above.

The workaround consists of using a previous working version of the atlassian-mail until the new one with the fix is released

1. Go to <bamboo-install-folder>/atlassian-bamboo/WEB-INF/lib/ and find the atlassian-mail-5.3.0.jar/atlassian-mail-5.2.1.jar version (depending on the version of Bamboo that you have).
2. Move the file outside the lib folder.
3. Download the file attached to this ticket called 'atlassian-mail-5.0.0.jar' and move it to the <bamboo-install-folder>/atlassian-bamboo/WEB-INF/lib/ directory.
4. Check that the file has the right/same permissions as the other files in the folder.
5. Restart Bamboo

For Bamboo versions 10.x and above you might want to try workaround described here: https://support.atlassian.com/bamboo/kb/mail-configuration-fails-after-application-upgrade-with-an-authenticationfailedexception/
Basically remove the username and password from the smtp configuration file and restart Bamboo.