Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-24856

When the quartz worker dies, there's no warning that services are no longer being run

    XMLWordPrintable

    Details

      Description

      We recently ran into a situation where a quartz worker thread died due to an error being thrown and services were not being run as a result. We should be able to catch the throwable and attempt to log that it happened so remediation action can be taken.

      An associated stack trace:

      011-06-16 10:23:34,563 QuartzWorker-0 ERROR ServiceRunner    Backup Service [jira.bc.dataimport.DefaultExportService] Error exporting Active Objects. You must run JRE 1.6_18 or higher. java.runtime.version: 1.6.0_12-b04
      java.lang.NoSuchMethodError: javax.xml.stream.XMLOutputFactory.newFactory(Ljava/lang/String;Ljava/lang/ClassLoader;)Ljavax/xml/stream/XMLOutputFactory;
      	at com.atlassian.dbexporter.node.stax.StaxUtils.newXmlOutputFactory(StaxUtils.java:189)
      	at com.atlassian.dbexporter.node.stax.StaxUtils.newXmlOutputFactory(StaxUtils.java:172)
      	at com.atlassian.dbexporter.node.stax.StaxStreamWriter.createXmlStreamWriter(StaxStreamWriter.java:53)
      	at com.atlassian.dbexporter.node.stax.StaxStreamWriter.<init>(StaxStreamWriter.java:39)
      	at com.atlassian.activeobjects.backup.ActiveObjectsBackup.save(ActiveObjectsBackup.java:105)
      	at com.atlassian.jira.bc.dataimport.DefaultExportService.export(DefaultExportService.java:91)
      	at com.atlassian.jira.bc.dataimport.DefaultExportService.export(DefaultExportService.java:53)
      	at com.atlassian.jira.service.services.export.ExportService.performBackup(ExportService.java:136)
      	at com.atlassian.jira.service.services.export.ExportService.run(ExportService.java:121)
      	at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60)
      	at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:195)
      	at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
      Exception in thread "QuartzWorker-0" java.lang.NoSuchMethodError: Error exporting Active Objects. You must run JRE 1.6_18 or higher. java.runtime.version: 1.6.0_12-b04
      	at com.atlassian.jira.bc.dataimport.DefaultExportService.export(DefaultExportService.java:99)
      	at com.atlassian.jira.bc.dataimport.DefaultExportService.export(DefaultExportService.java:53)
      	at com.atlassian.jira.service.services.export.ExportService.performBackup(ExportService.java:136)
      	at com.atlassian.jira.service.services.export.ExportService.run(ExportService.java:121)
      	at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60)
      	at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:195)
      

      caused services not to be run.

      In short, a scheduled job that throws anything but a RuntimeException can kill a Quartz scheduler worker thread with no means of recovery. Even a single bad job running every few minutes can completely exhaust the pool of workers before long

      Verification

      Generate a Thread Dump and check it to see if there are any QuartzWorker threads running.

      Workaround

      1. Manually flush the mail queue. This must be done as the mail queue is not persisted over restarts - there is a feature request for this under JRA-4665.
      2. Stop JIRA.
      3. Replace JIRA Installation Directory\atlassian-jira\WEB-INF\lib\quartz-1.5.1.jar with quartz-1.5.1-atlassian-2.jar.
      4. Start JIRA.

      If you have not identified what's throwing the exception, and fixing it (or disabling it), it is likely that the scheduler may crash again. Reviewing the logs to see if any scheduled jobs are throwing anything but a RuntimeException will identify possible causes.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              ohernandez@atlassian.com Oswaldo Hernandez (Inactive)
              Reporter:
              mtan MinanA
              Votes:
              5 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: