When user clicks the "Test" button on the SMTP server settings page, values entered by user are immediately persisted in bamboo-mail.cfg.xml file. As a result, clicking "Cancel" link after "Test" will not restore original settings. 

In BDEV-12340, implemented in 5.16.0+ the "Test" button has been renamed to "Save and test" to make it more clear how it works.

However, this is just a workaround, we need a solution in which the "Test" button will use data entered in UI, without persisting it.