Lock starvation for mail handlers using the same mailbox

Summary

JRASERVER-65581 introduced cluster locking for message processing. MailFetcherService is locking on MailServer id and the folder of that mailbox that is being processed.
This is how the introduced code looks like:

        final String lockName = MailFetcherService.class.getName() + "." + mailServer.getId() + "." + getFolderName(mailServer);
        final ClusterLock lock = clusterLockService.getLockForName(lockName);

        if (!lock.tryLock()) {
            log.debug(String.format("Unable to acquire %s lock. Skipping the run.", lockName));
            return;
        }

For instances that have many mail handlers configured to use the same Mail Server and run with the same interval they will compete for the lock and some of them may starve.

Symptoms

Some of the mail handlers are not executed.
Example log messages that may indicate symptoms of this issue:
JIRA mail logs would keep showing the following stack trace for multiple times:

2017-11-08 17:00:00,089 DEBUG [JIRA IMAP] Caesium-1-3 anonymous    test Unable to acquire com.atlassian.jira.service.services.mail.MailFetcherService.10001.INBOX lock. Skipping the run.
2017-11-08 17:00:00,090 DEBUG [JIRA IMAP] Caesium-1-1 anonymous    test 2 Unable to acquire com.atlassian.jira.service.services.mail.MailFetcherService.10001.INBOX lock. Skipping the run.
2017-11-08 17:00:00,092 DEBUG [JIRA IMAP] Caesium-1-4 anonymous    test 3 Unable to acquire com.atlassian.jira.service.services.mail.MailFetcherService.10001.INBOX lock. Skipping the run.
2017-11-08 17:00:00,093 DEBUG [JIRA IMAP] Caesium-1-3 anonymous    test 4 Unable to acquire com.atlassian.jira.service.services.mail.MailFetcherService.10001.INBOX lock. Skipping the run.
2017-11-08 17:00:00,093 DEBUG [JIRA IMAP] Caesium-1-1 anonymous    test 5 Unable to acquire com.atlassian.jira.service.services.mail.MailFetcherService.10001.INBOX lock. Skipping the run.

This is the same name, for all importers. So if one of the 120 Mail Handlers is running, the others are blocked, due to the fact, the current one running blocks all the others.

Expected behaviour

Locking on different level of granularity (for example on the mail service instance) or fair locking so that all handlers are called eventually.

Workarounds

• If possible try to categorise the messages on mail box level (filter or rules) and put them in different folders. Reconfigure mail handlers to use dedicated folders. This only works with IMAP mail boxes.
• Add a new Mail Server (every one of them can point to the same physical mailbox) in Jira for each of your Mail Handlers then associate every one of your Mail Handler with a dedicated Mail Server but you may observe symptoms of JRASERVER-65581.