We haven't seen it since upgrading to 7.4.6

Hi,

If we plan to upgrade to version 7.6.7 Enterprise Edition, will it still have this bug occur? Could you advise? Thanks!

Hi everyone.
Just want to let you know Jira 7.3.9 and 7.7.0 that contain the fix is released on 11 Jan 2018.

7.4.6, 7.5.4 versions are releasing soon.

Cheers,
Ignat Alexeyenko
Jira bugmaster.

Hi,

Just wanted to let you know that Jira 7.6.2 which contains the fix has been released today. 
7.3.9, 7.4.6, 7.5.4 are planned for January 2018, yet don't please treat the disclosed release plans as time commitment as things may slip.

The fix for other mentioned releases would be available too, yet it's too early to say when exactly.

Cheers,
Ignat
Jira bugmaster.

lwlodarczyk - That's great news!  Thanks!

I'm sorry for the miscommunication around the status of this issue. Moving it back to Awaiting Development was a mistake and then it escaped our filters.

This issue has been fixed and will be rolled out in all subsequent releases for versions 7.2+. The earliest expected deployments are 7.6.2 and 7.2.13.

I see the UIS and Support reference count going up.  I also see it was moved to In Progress a couple days ago, but only lasted there about 15 minutes before it was moved back to Awaiting Development.  Do we know when this one is planned?  It's a high concern for our enterprise solution as it's already happened to us once.  The workaround provided did resolve the issue, but we're concerned it could occur  again and would like to mitigate that risk.  Thanks!

HI Support,

When the error " multiple entry with same key "happened at our environment as below, we notice outgoing email from Service Desk project type will not working (user wont receive outgoing email notification). Until the duplicate entry been remove, and Jira restart, it will just resolve the issue.
java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=
Our Jira is using globally by every different site, so when we meet this error, we will try to minimal the downtime, and schedule the restart during lunch time or after office hour . Would like to check what is the outgoing mail queue length where the notifications are queuing up? Our concern is we don't know when Jira outgoing is running out of queue and cause message lost. Please advise. Thanks!

Hi

We confired "Steps to Reproduce".
We guess that "Auto-close after being resolved for 3 bussiness days" function connect with JRASERVER-64325

Steps to Reproduce
---------------------
1.We create Service Desk Project of IT Service Desk Type.
("Auto-close after being resolved for 3 bussiness days" function is created in Service Desk Project of IT Service Desk Type)

2.We request in Potal Screen.
3.Service desk Agent change status of request to resolved status at 2017/11/14.
4.We confirm that "Time to close after resolution" of request display 23:56.
5.We change date of OS to future date.

date -s "11/20 09:00 2017"
hwclock -w

6.We restart JIRA service twice.
7.we confrim that the folloing Error.

2017-11-20 09:10:33,407 JIRA-Bootstrap ERROR      [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.			
java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=Mon Nov 20 09:05:35 JST 2017 and sd.custom.notification.batch.send=Mon Nov 20 09:05:35 JST 2017			
        at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)			
        at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)			
        at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70)			
        at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254)			
        at com.atlassian.jira.scheduler.OfBizClusteredJobDao.refresh(OfBizClusteredJobDao.java:111)			
        at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsFromDao(SchedulerQueueImpl.java:139)			
        at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsUnderLock(SchedulerQueueImpl.java:130)			
        at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobs(SchedulerQueueImpl.java:117)			
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.refreshClusteredJobs(CaesiumSchedulerService.java:359)			
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.startImpl(CaesiumSchedulerService.java:275)

---------------------

Ricksoft

Bartek - how many issues do your rules typically process? We only have a single rule, and this seems to be a weekly occurrence now. The option you mentioned is also unchecked. 

Jarrod, if you're using Automation I suggest to uncheck option Process all issues produced by this trigger in bulk. It seems that in recent versions of the plugin it is much less prone to clustered jobs failure when this option isn't used. I also scheduled all rules to launch at separated times to be sure that they don't collide with each other.

During last couple of months we had only one clustered jobs failure, so it seems that it is a lot better now

It's happened to me before we had the automation plug-in too. It happened once about six months ago. Now, we installed the plugin, and it's happened twice in two weeks. I'm not saying that it's exactly what's causing it, but it seems related. 

It's not related to the Automation plugins. We don't use them, nor have them installed for that matter.

We just implemented Automation Lite for JIRA and it's happened to us twice in two weeks. Only happened one other time over the past 6 months or so we have had JIRA SD. 

As a small workaround to help us when this happens (we have a very large JIRA instance supporting worldwide users and can't always take it down immediately for maintenance), we change the JIRA Service Desk configuration in Applications (https://jiralocation/secure/admin/SDConfiguration.jspa) to send both SD and regular notifications.

It's not ideal since that means project admins can no longer directly control their notifications or take advantage of SD features, but it helps a bit.

Hi,
with us it was presumably the JOB sd.custom.notification.batch.clean.up, which ran exactly at the time, to the double entry in the table custeredjob with the JOB ID sd.custom.notification.batch.send is.

Best regards
Dieter Pohl

We have the same recurring problem in JIRA 7.3.3. disabling our Service Desk notifications once in a while. Anyone found out what causes this problem? I am currently suspecting Automation Lite for JIRA plugin because the problem occurs randomly but always couple of minutes after some scheduled rule from Automation was launched in our biggest Service Desk project.

Is there anyone here that doesn't use Automation for JIRA but still experiences this problem? Especially the part with Service Desk notifications being stuck?

Since this isn't a resolved issue yet, here is how we're now monitoring this so if it happens again we'll catch it within a few minutes. We're using Zabbix for monitoring.

userparameter_psql.conf

UserParameter=psql.clusteredjob,PGPASSWORD=(REDACTED_PASSWD) psql -U (REDACTED_USER) -t -d jira -h (REDACTED_HOST) -c "SELECT * FROM clusteredjob WHERE job_id in (SELECT job_id FROM clusteredjob GROUP BY job_id HAVING COUNT(*)>1)" | wc -l

This will return the number of characters for the above psql query. Which will typically be "1" if there is no duplicate job_ids. If there is duplicate rows, the above will return a value >1.

Then within zabbix triggers for the host, I am checking against the expression:

{(REDACTED_HOST):psql.clusteredjob.last()}>1

If the trigger detects a value greater than (>) 1, it will trigger a High severity trigger for us.

+1 for me with same issue.

This duplicate job ID stopped all outbound processing of our JSD emails and prevented JIRA from starting up when restarted.

JIRA: 7.3.6
JSD: 3.5.0
DB: PostgreSQL 9.5

Error:

java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=Sat Aug 19 08:53:29 AEST 2017 and sd.custom.notification.batch.send=Sat Aug 19 08:53:29 AEST 2017

The fix for us was to remove the duplicate job from the JD, which I found by running:

select * from clusteredjob where job_id = 'sd.custom.notification.batch.send';

Once removed, all outbound emails were processed and successfully delivered to our customers.

You guys really need to fix this one quickly, it is causing major issues for us and I'm sure for many others that haven't even found it yet.  We have to continually delete the extra rows from the db so that our notifications go out.  Can we get an update please?

Yesterday I got a report from my JIRA Service Desk (3.6.1) customers that emails from my Service Desk Agents are not generating conversation emails. After checking my mail server used for sending the notifications, I found out that email conversations were not going out to the customers for 3 days! Even though the service desk troubleshooting gave no signals of anything working incorrectly. Even page with JIRA scheduler details showed only green lights for every service desk job. Turning on the debug level for outgoing emails also showed nothing.

While every light in the JIRA health-check is green, in the meantime, something in the background completely broke JIRA Service Desk's email communication (which is the backbone of the JIRA Service Desk support process).

Finding no source of the error I have restarted our JIRA Software (7.4.1) instance and to my surprise, JIRA failed to start with error:

java.lang.IllegalArgumentException: *Multiple entries with same key: sd.custom.notification.batch.send*=Tue Aug 01 00:02:01 CEST 2017 and sd.custom.notification.batch.send=Tue Aug 01 00:02:01 CEST 2017 at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150) at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104) at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70) at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254) at com.atlassian.jira.scheduler.OfBizClusteredJobDao.refresh(OfBizClusteredJobDao.java:111) at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsFromDao(SchedulerQueueImpl.java:139) at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsUnderLock(SchedulerQueueImpl.java:130) at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobs(SchedulerQueueImpl.java:117) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.refreshClusteredJobs(CaesiumSchedulerService.java:359) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.startImpl(CaesiumSchedulerService.java:275) at com.atlassian.jira.scheduler.JiraCaesiumSchedulerService.startImpl(JiraCaesiumSchedulerService.java:36) at com.atlassian.scheduler.core.AbstractSchedulerService.start(AbstractSchedulerService.java:180) at com.atlassian.jira.scheduler.JiraSchedulerLauncher.proceedIfAllClear(JiraSchedulerLauncher.java:37) at com.atlassian.jira.scheduler.JiraSchedulerLauncher.start(JiraSchedulerLauncher.java:23) at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:56) at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:181) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:155) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:304) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:199) at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:146) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:105) at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31) at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:103) at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:157) at java.lang.Thread.run(Thread.java:745)

After applying the workaround by finding and deleting the duplicate entry from "clusteredjob" table I have initiated JIRA restart. Thankfully, everything went smooth and every customer email which was stuck for last 3 days, was generated in the JIRA mail queue.

My support team had 3 days of tranquility but after fixing this issue, the reality has caught up 

It is not the first time that I encounter issues with JIRA Service Desk notifications not going out this is why I have a feature suggestion to prevent such situations:

JIRA Service Desk customer notification audit,  where at least we could see the frequency of outgoing customer emails and maybe even, set alarm rules for situations when no customer notification is generated for the whole day.

If such alarms existed, any JIRA Service Desk manager would know about issues with customer emails after one day instead of three (or more).

Confirmed that the workaround works for JIRA Software 7.2.8 Data Center / JIRA Service Desk 3.2.8 Data Center as well.

i just ran into this with JIRA 7.2.8. The workaround worked!