Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-64325

Jira fails to start due to duplicate job IDs in the clusteredjob table

    • 7.01
    • 55
    • Severity 1 - Critical
    • 1,303
    • Hide
      Atlassian Update – 12 January 2018

      Hi everyone,

      All the Jira versions, which have the bug fixed, are now released.
      Wish you a smooth upgrade.

      Cheers,
      Ignat Alexeyenko
      Jira bugmaster.

      Show
      Atlassian Update – 12 January 2018 Hi everyone, All the Jira versions, which have the bug fixed, are now released. Wish you a smooth upgrade. Cheers, Ignat Alexeyenko Jira bugmaster.

      Summary

      Caesium method getJobDetails calls clusteredJobDao.find(jobId) and expects single uniq value from DB, while clusteredjob table does not ensure this.

      Steps to Reproduce

      1. Create duplicated records in table
      2. Restart JIRA

      Expected Results

      JIRA works fine

      Actual Results

      JIRA is failing to start with the following errors:

      2017-03-03 02:26:53,599 Structure-Jobs-1143be31 Queue-Thread#1 ERROR anonymous     [c.a.c.manager.application.ApplicationServiceGeneric] com.atlassian.cache.CacheException: java.lang.IllegalStateException: Too many rows found for query on ClusteredJob
          	row1: OfBizClusteredJob[id=81932,delegate=ImmutableClusteredJob[jobId=com.atlassian.jira.crowd.embedded.JiraDirectoryPollerManager.10001,jobRunnerKey=com.atlassian.jira.crowd.embedded.JiraDirectoryPollerManager,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Mar 03 01:56:51 PST 2017,intervalInMillis=3540000]],nextRunTime=Fri Mar 03 01:56:51 PST 2017,version=1,rawParameters=[ .. ]
      row2:
      	OfBizClusteredJob[id=82235,delegate=ImmutableClusteredJob[jobId=com.atlassian.jira.crowd.embedded.JiraDirectoryPollerManager.10001,jobRunnerKey=com.atlassian.jira.crowd.embedded.JiraDirectoryPollerManager,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Mar 03 01:56:51 PST 2017,intervalInMillis=3540000]],nextRunTime=Fri Mar 03 01:56:51 PST 2017,version=1,rawParameters=[ .. ]]
      com.atlassian.crowd.exception.DirectoryInstantiationException: com.atlassian.cache.CacheException: java.lang.IllegalStateException: Too many rows found for query on 
      	at com.atlassian.crowd.directory.loader.CacheableDirectoryInstanceLoader.getDirectory(CacheableDirectoryInstanceLoader.java:92)
      	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.getDirectoryImplementation(DirectoryManagerGeneric.java:272)
      ....
      Caused by: com.atlassian.cache.CacheException: java.lang.IllegalStateException: Too many rows found for query on ClusteredJob
      	row1: OfBizClusteredJob[id=81932,delegate=ImmutableClusteredJob[jobId=com.atlassian.jira.crowd.embedded.JiraDirectoryPollerManager.10001,jobRunnerKey=com.atlassian.jira.crowd.embedded.JiraDirectoryPollerManager,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Mar 03 01:56:51 PST 2017,intervalInMillis=3540000]],nextRunTime=Fri Mar 03 01:56:51 PST 2017,version=1,rawParameters=
      row2:
      	OfBizClusteredJob[id=82235,delegate=ImmutableClusteredJob[jobId=com.atlassian.jira.crowd.embedded.JiraDirectoryPollerManager.10001,jobRunnerKey=com.atlassian.jira.crowd.embedded.JiraDirectoryPollerManager,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Mar 03 01:56:51 PST 2017,intervalInMillis=3540000]],nextRunTime=Fri Mar 03 01:56:51 PST 2017,version=1,rawParameters=
      	at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl$1.consume(SelectQueryImpl.java:198)
      	at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:231)
      	at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.consumeWith(SelectQueryImpl.java:214)
      	at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.singleValue(SelectQueryImpl.java:191)
      	at com.atlassian.jira.scheduler.OfBizClusteredJobDao.find(OfBizClusteredJobDao.java:88)
      	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.getJobDetails(CaesiumSchedulerService.java:230)
      	at com.atlassian.scheduler.core.DelegatingSchedulerService.getJobDetails(DelegatingSchedulerService.java:97)
      	at com.atlassian.jira.crowd.embedded.JiraDirectoryPollerManager.hasPoller(JiraDirectoryPollerManager.java:64)
      	at com.atlassian.crowd.manager.directory.monitor.DirectoryMonitorManagerImpl.hasMonitor(DirectoryMonitorManagerImpl.java:108)
      	at com.atlassian.crowd.directory.loader.DbCachingRemoteDirectoryInstanceLoader.getDirectory(DbCachingRemoteDirectoryInstanceLoader.java:113)
      ...
      

      Notes

      Related Indexes for clusteredjob table:

      • "pk_clusteredjob" PRIMARY KEY, btree (id)
      • "clusteredjob_jobid_idx" btree (job_id)

      Workaround

            [JRASERVER-64325] Jira fails to start due to duplicate job IDs in the clusteredjob table

            Hi hploh,

            The bug fixed since 7.6.2 onwards. So the bug would not occur in 7.6.7 ER version you plan to upgrade.
            Wish you a smooth upgrade!

            Cheers,
            Ignat Alexeyenko
            Jira bugmaster

            Ignat (Inactive) added a comment - Hi hploh , The bug fixed since 7.6.2 onwards. So the bug would not occur in 7.6.7 ER version you plan to upgrade. Wish you a smooth upgrade! Cheers, Ignat Alexeyenko Jira bugmaster

            We haven't seen it since upgrading to 7.4.6

            James E. Hunt [ASRC Federal] added a comment - We haven't seen it since upgrading to 7.4.6

            HP added a comment -

            Hi,

            If we plan to upgrade to version 7.6.7 Enterprise Edition, will it still have this bug occur? Could you advise? Thanks!

            HP added a comment - Hi, If we plan to upgrade to version 7.6.7 Enterprise Edition, will it still have this bug occur? Could you advise? Thanks!

            Hi everyone.
            Just want to let you know Jira 7.3.9 and 7.7.0 that contain the fix is released on 11 Jan 2018.

            7.4.6, 7.5.4 versions are releasing soon.

            Cheers,
            Ignat Alexeyenko
            Jira bugmaster.

            Ignat (Inactive) added a comment - Hi everyone. Just want to let you know Jira 7.3.9 and 7.7.0 that contain the fix is released on 11 Jan 2018. 7.4.6, 7.5.4 versions are releasing soon. Cheers, Ignat Alexeyenko Jira bugmaster.

            Hi,

            Just wanted to let you know that Jira 7.6.2 which contains the fix has been released today. 
            7.3.9, 7.4.6, 7.5.4 are planned for January 2018, yet don't please treat the disclosed release plans as time commitment as things may slip.

            The fix for other mentioned releases would be available too, yet it's too early to say when exactly.

            Cheers,
            Ignat
            Jira bugmaster.

            Ignat (Inactive) added a comment - Hi, Just wanted to let you know that Jira 7.6.2 which contains the fix has been released today.  7.3.9, 7.4.6, 7.5.4 are planned for January 2018, yet don't please treat the disclosed release plans as time commitment as things may slip. The fix for other mentioned releases would be available too, yet it's too early to say when exactly. Cheers, Ignat Jira bugmaster.

            lwlodarczyk - That's great news!  Thanks!

            Carol Jones added a comment - lwlodarczyk - That's great news!  Thanks!

            I'm sorry for the miscommunication around the status of this issue. Moving it back to Awaiting Development was a mistake and then it escaped our filters.

            This issue has been fixed and will be rolled out in all subsequent releases for versions 7.2+. The earliest expected deployments are 7.6.2 and 7.2.13.

            Lukasz Wlodarczyk added a comment - I'm sorry for the miscommunication around the status of this issue. Moving it back to Awaiting Development was a mistake and then it escaped our filters. This issue has been fixed and will be rolled out in all subsequent releases for versions 7.2+. The earliest expected deployments are 7.6.2 and 7.2.13.

            I see the UIS and Support reference count going up.  I also see it was moved to In Progress a couple days ago, but only lasted there about 15 minutes before it was moved back to Awaiting Development.  Do we know when this one is planned?  It's a high concern for our enterprise solution as it's already happened to us once.  The workaround provided did resolve the issue, but we're concerned it could occur  again and would like to mitigate that risk.  Thanks!

            Carol Jones added a comment - I see the UIS and Support reference count going up.  I also see it was moved to In Progress a couple days ago, but only lasted there about 15 minutes before it was moved back to Awaiting Development.  Do we know when this one is planned?  It's a high concern for our enterprise solution as it's already happened to us once.  The workaround provided did resolve the issue, but we're concerned it could occur  again and would like to mitigate that risk.  Thanks!

            HP added a comment -

            HI Support,

            When the error " multiple entry with same key "happened at our environment as below, we notice outgoing email from Service Desk project type will not working (user wont receive outgoing email notification). Until the duplicate entry been remove, and Jira restart, it will just resolve the issue.
            java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=
            Our Jira is using globally by every different site, so when we meet this error, we will try to minimal the downtime, and schedule the restart during lunch time or after office hour . Would like to check what is the outgoing mail queue length where the notifications are queuing up? Our concern is we don't know when Jira outgoing is running out of queue and cause message lost. Please advise. Thanks!

            HP added a comment - HI Support, When the error " multiple entry with same key "happened at our environment as below, we notice outgoing email from Service Desk project type will not working (user wont receive outgoing email notification). Until the duplicate entry been remove, and Jira restart, it will just resolve the issue. java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send= Our Jira is using globally by every different site, so when we meet this error, we will try to minimal the downtime, and schedule the restart during lunch time or after office hour . Would like to check what is the outgoing mail queue length where the notifications are queuing up? Our concern is we don't know when Jira outgoing is running out of queue and cause message lost. Please advise. Thanks!

            Hi

            We confired "Steps to Reproduce".
            We guess that "Auto-close after being resolved for 3 bussiness days" function connect with JRASERVER-64325

            Steps to Reproduce
            ---------------------
            1.We create Service Desk Project of IT Service Desk Type.
            ("Auto-close after being resolved for 3 bussiness days" function is created in Service Desk Project of IT Service Desk Type)

            2.We request in Potal Screen.
            3.Service desk Agent change status of request to resolved status at 2017/11/14.
            4.We confirm that "Time to close after resolution" of request display 23:56.
            5.We change date of OS to future date.

            date -s "11/20 09:00 2017"
            hwclock -w
            

            6.We restart JIRA service twice.
            7.we confrim that the folloing Error.

            2017-11-20 09:10:33,407 JIRA-Bootstrap ERROR      [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.			
            java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=Mon Nov 20 09:05:35 JST 2017 and sd.custom.notification.batch.send=Mon Nov 20 09:05:35 JST 2017			
                    at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)			
                    at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)			
                    at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70)			
                    at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254)			
                    at com.atlassian.jira.scheduler.OfBizClusteredJobDao.refresh(OfBizClusteredJobDao.java:111)			
                    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsFromDao(SchedulerQueueImpl.java:139)			
                    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsUnderLock(SchedulerQueueImpl.java:130)			
                    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobs(SchedulerQueueImpl.java:117)			
                    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.refreshClusteredJobs(CaesiumSchedulerService.java:359)			
                    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.startImpl(CaesiumSchedulerService.java:275)
            

            ---------------------

            Ricksoft

            Ricksoft Co., Ltd. added a comment - Hi We confired "Steps to Reproduce". We guess that "Auto-close after being resolved for 3 bussiness days" function connect with JRASERVER-64325 Steps to Reproduce --------------------- 1.We create Service Desk Project of IT Service Desk Type. ("Auto-close after being resolved for 3 bussiness days" function is created in Service Desk Project of IT Service Desk Type) 2.We request in Potal Screen. 3.Service desk Agent change status of request to resolved status at 2017/11/14. 4.We confirm that "Time to close after resolution" of request display 23:56. 5.We change date of OS to future date. date -s "11/20 09:00 2017" hwclock -w 6.We restart JIRA service twice. 7.we confrim that the folloing Error. 2017-11-20 09:10:33,407 JIRA-Bootstrap ERROR [c.a.jira.startup.LauncherContextListener] Unable to start JIRA. java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=Mon Nov 20 09:05:35 JST 2017 and sd.custom.notification.batch.send=Mon Nov 20 09:05:35 JST 2017 at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150) at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104) at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70) at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254) at com.atlassian.jira.scheduler.OfBizClusteredJobDao.refresh(OfBizClusteredJobDao.java:111) at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsFromDao(SchedulerQueueImpl.java:139) at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsUnderLock(SchedulerQueueImpl.java:130) at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobs(SchedulerQueueImpl.java:117) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.refreshClusteredJobs(CaesiumSchedulerService.java:359) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.startImpl(CaesiumSchedulerService.java:275) --------------------- Ricksoft

              pczuj Przemyslaw Czuj
              ayakovlev@atlassian.com Andriy Yakovlev [Atlassian]
              Affected customers:
              67 This affects my team
              Watchers:
              84 Start watching this issue

                Created:
                Updated:
                Resolved: