Details
-
Bug
-
Resolution: Fixed
-
High
-
4.4, 5.2.2, 6.1.1
-
4.04
-
Description
We recently ran into a situation where a quartz worker thread died due to an error being thrown and services were not being run as a result. We should be able to catch the throwable and attempt to log that it happened so remediation action can be taken.
An associated stack trace:
011-06-16 10:23:34,563 QuartzWorker-0 ERROR ServiceRunner Backup Service [jira.bc.dataimport.DefaultExportService] Error exporting Active Objects. You must run JRE 1.6_18 or higher. java.runtime.version: 1.6.0_12-b04 java.lang.NoSuchMethodError: javax.xml.stream.XMLOutputFactory.newFactory(Ljava/lang/String;Ljava/lang/ClassLoader;)Ljavax/xml/stream/XMLOutputFactory; at com.atlassian.dbexporter.node.stax.StaxUtils.newXmlOutputFactory(StaxUtils.java:189) at com.atlassian.dbexporter.node.stax.StaxUtils.newXmlOutputFactory(StaxUtils.java:172) at com.atlassian.dbexporter.node.stax.StaxStreamWriter.createXmlStreamWriter(StaxStreamWriter.java:53) at com.atlassian.dbexporter.node.stax.StaxStreamWriter.<init>(StaxStreamWriter.java:39) at com.atlassian.activeobjects.backup.ActiveObjectsBackup.save(ActiveObjectsBackup.java:105) at com.atlassian.jira.bc.dataimport.DefaultExportService.export(DefaultExportService.java:91) at com.atlassian.jira.bc.dataimport.DefaultExportService.export(DefaultExportService.java:53) at com.atlassian.jira.service.services.export.ExportService.performBackup(ExportService.java:136) at com.atlassian.jira.service.services.export.ExportService.run(ExportService.java:121) at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60) at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47) at org.quartz.core.JobRunShell.run(JobRunShell.java:195) at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520) Exception in thread "QuartzWorker-0" java.lang.NoSuchMethodError: Error exporting Active Objects. You must run JRE 1.6_18 or higher. java.runtime.version: 1.6.0_12-b04 at com.atlassian.jira.bc.dataimport.DefaultExportService.export(DefaultExportService.java:99) at com.atlassian.jira.bc.dataimport.DefaultExportService.export(DefaultExportService.java:53) at com.atlassian.jira.service.services.export.ExportService.performBackup(ExportService.java:136) at com.atlassian.jira.service.services.export.ExportService.run(ExportService.java:121) at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60) at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47) at org.quartz.core.JobRunShell.run(JobRunShell.java:195)
caused services not to be run.
In short, a scheduled job that throws anything but a RuntimeException can kill a Quartz scheduler worker thread with no means of recovery. Even a single bad job running every few minutes can completely exhaust the pool of workers before long
Verification
Generate a Thread Dump and check it to see if there are any QuartzWorker threads running.
Workaround
- Manually flush the mail queue. This must be done as the mail queue is not persisted over restarts - there is a feature request for this under JRA-4665.
- Stop JIRA.
- Replace JIRA Installation Directory\atlassian-jira\WEB-INF\lib\quartz-1.5.1.jar with quartz-1.5.1-atlassian-2.jar
. - Start JIRA.
If you have not identified what's throwing the exception, and fixing it (or disabling it), it is likely that the scheduler may crash again. Reviewing the logs to see if any scheduled jobs are throwing anything but a RuntimeException will identify possible causes.
Attachments
Issue Links
- relates to
-
JRASERVER-4665 Need to persist Mail Queue Items
- Gathering Interest
- was cloned as
-
JDEV-26170 Loading...