Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-39825

Quartz scheduled jobs can stop running until restart

XMLWordPrintable

      Database request is performed before each schedule job is run. If this request fails for any reason (e.g. connection timeout, not enough DB connections in the pool, table missing in DB), the scheduled job will be prevented from running until the instance is restarted.

      This is the relevant stack trace:

      org.quartz.SchedulerException: JobListener 'ScheduledJobListener' threw exception: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection [See nested exception: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection]
	at org.quartz.core.QuartzScheduler.notifyJobListenersToBeExecuted(QuartzScheduler.java:1951)
	at org.quartz.core.JobRunShell.notifyListenersBeginning(JobRunShell.java:364)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:190)
	at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool.lambda$runInThread$187(ConfluenceQuartzThreadPool.java:19)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection
	at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:473)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374)
	at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
	at com.sun.proxy.$Proxy33.getTransaction(Unknown Source)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at com.sun.proxy.$Proxy142.jobToBeExecuted(Unknown Source)
	at org.quartz.core.QuartzScheduler.notifyJobListenersToBeExecuted(QuartzScheduler.java:1949)
	... 4 more
Caused by: net.sf.hibernate.exception.GenericJDBCException: Cannot open connection
	at net.sf.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:80)
	at net.sf.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:69)
	at net.sf.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
	at net.sf.hibernate.impl.BatcherImpl.convert(BatcherImpl.java:328)
	at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:295)
	at net.sf.hibernate.impl.SessionImpl.connection(SessionImpl.java:3367)
	at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422)
	... 17 more
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:126)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
	at net.sf.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:59)
	at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:292)
	... 20 more
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
	... 23 more

      Two job listeners are called when execution is starting:

      1. ExecutingJobsManager keeps a list of jobs which are executing
      2. ScheduledJobsListener saves execution details to database

      If ScheduledJobsListener throws an exception, ExecutingJobsManager will not get notified that the execution got cancelled. Further executions get cancelled because AbstractJob#executeInternal(..) checks ExecutingJobsManager to see if similar (but not same) execution is already happening.

      Workaround for Confluence 5.8

      For users of Confluence 5.8.18, we are providing a patch file containing a fix for this issue. Please be aware that this patch has not been fully tested and is applied at your own risk. We strongly recommend that you test the patch on a staging instance of Confluence before applying it to a production instance. Refer to https://confluence.atlassian.com/doc/installing-patched-class-files-166528493.html for details on how to apply the patch. Also, make sure that the MaxWaitMillis is set to 10000 so that the connection pool got sometime to recover.

        1. confluence-5.8.18-CONF-39825-patch.zip
          11 kB
        2. Снимок экрана от 2017-04-04 15-09-06.png
          Снимок экрана от 2017-04-04 15-09-06.png
          239 kB

              jmcarthur James McArthur (Inactive)
              onevalainen Olli Nevalainen
              Votes:
              16 Vote for this issue
              Watchers:
              49 Start watching this issue

                Created:
                Updated:
                Resolved: