Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-39825

Quartz scheduled jobs can stop running until restart

XMLWordPrintable

      Database request is performed before each schedule job is run. If this request fails for any reason (e.g. connection timeout, not enough DB connections in the pool, table missing in DB), the scheduled job will be prevented from running until the instance is restarted.

      This is the relevant stack trace:

      org.quartz.SchedulerException: JobListener 'ScheduledJobListener' threw exception: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection [See nested exception: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection]
	at org.quartz.core.QuartzScheduler.notifyJobListenersToBeExecuted(QuartzScheduler.java:1951)
	at org.quartz.core.JobRunShell.notifyListenersBeginning(JobRunShell.java:364)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:190)
	at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool.lambda$runInThread$187(ConfluenceQuartzThreadPool.java:19)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection
	at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:473)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374)
	at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
	at com.sun.proxy.$Proxy33.getTransaction(Unknown Source)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at com.sun.proxy.$Proxy142.jobToBeExecuted(Unknown Source)
	at org.quartz.core.QuartzScheduler.notifyJobListenersToBeExecuted(QuartzScheduler.java:1949)
	... 4 more
Caused by: net.sf.hibernate.exception.GenericJDBCException: Cannot open connection
	at net.sf.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:80)
	at net.sf.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:69)
	at net.sf.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
	at net.sf.hibernate.impl.BatcherImpl.convert(BatcherImpl.java:328)
	at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:295)
	at net.sf.hibernate.impl.SessionImpl.connection(SessionImpl.java:3367)
	at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422)
	... 17 more
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:126)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
	at net.sf.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:59)
	at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:292)
	... 20 more
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
	... 23 more

      Two job listeners are called when execution is starting:

      1. ExecutingJobsManager keeps a list of jobs which are executing
      2. ScheduledJobsListener saves execution details to database

      If ScheduledJobsListener throws an exception, ExecutingJobsManager will not get notified that the execution got cancelled. Further executions get cancelled because AbstractJob#executeInternal(..) checks ExecutingJobsManager to see if similar (but not same) execution is already happening.

      Workaround for Confluence 5.8

      For users of Confluence 5.8.18, we are providing a patch file containing a fix for this issue. Please be aware that this patch has not been fully tested and is applied at your own risk. We strongly recommend that you test the patch on a staging instance of Confluence before applying it to a production instance. Refer to https://confluence.atlassian.com/doc/installing-patched-class-files-166528493.html for details on how to apply the patch. Also, make sure that the MaxWaitMillis is set to 10000 so that the connection pool got sometime to recover.

        1. confluence-5.8.18-CONF-39825-patch.zip
          11 kB
        2. Снимок экрана от 2017-04-04 15-09-06.png
          Снимок экрана от 2017-04-04 15-09-06.png
          239 kB

          Form Name

            [CONFSERVER-39825] Quartz scheduled jobs can stop running until restart

            Dmitry Bod added a comment -

            Hello David! 

            I have already fixed it today but I did create an issue! 

            By the way, here it is https://getsupport.atlassian.com/servicedesk/customer/portal/14/CSP-200534

             

            Best wishes!

            Dmitry Bod added a comment - Hello David!  I have already fixed it today but I did create an issue!  By the way, here it is https://getsupport.atlassian.com/servicedesk/customer/portal/14/CSP-200534   Best wishes!

            David Mason (Inactive) added a comment -

            Hi diboan.spb1157675044,

            It looks like you might be seeing a different issue. If you could raise a support case with us at https://support.atlassian.com/contact/, a support engineer will be able to help you with this.

            Thanks,
            David Mason

            David Mason (Inactive) added a comment - Hi diboan.spb1157675044 , It looks like you might be seeing a different issue. If you could raise a support case with us at https://support.atlassian.com/contact/ , a support engineer will be able to help you with this. Thanks, David Mason

            Dmitry Bod added a comment - - edited

            Hello there! Does anybody still watching this issue? 

            I did as you all said, but I have 500 error 

             

            My confluence version 5.9.11

            Dmitry Bod added a comment - - edited Hello there! Does anybody still watching this issue?  I did as you all said, but I have 500 error    My confluence version 5.9.11

            Minh Tran added a comment -

            Hi vbrimmer,

            Yes we don't need to patch it when upgrading to 6.0.3

            Thanks,
            Minh Tran
            Confluence BugMaster

            Minh Tran added a comment - Hi vbrimmer , Yes we don't need to patch it when upgrading to 6.0.3 Thanks, Minh Tran Confluence BugMaster

            Scottie Brimmer added a comment -

            So I patched 5.8.x and now am upgrading that instance to 6.0.3.  Is it safe to assume that I do not have to go back in and re-add any of the files that were included in the patch .zip?  They show as "added files" in the "5.8.13 modifications" file that the wizard produced.

            Scottie Brimmer added a comment - So I patched 5.8.x and now am upgrading that instance to 6.0.3.  Is it safe to assume that I do not have to go back in and re-add any of the files that were included in the patch .zip?  They show as "added files" in the "5.8.13 modifications" file that the wizard produced.

            Peter Binney added a comment -

            Thank you, though I assume I do not deploy the __MACOSX directory and .DS_Store files in the .zip
            Having deployed the xml and the two .class files, Confluence now starts.
            Could I suggest that you fix your patch deployment page to cover this.

            I can confirm that the patch has fixed the problem. For your reference, the steps I use to trigger it are:
            1. Run: ALTER DATABASE confluence set single_user with rollback immediate
            2. Try to view a page - which hangs or errors as expected
            3. Run: ALTER DATABASE confluence set multi_user
            4. Edit a page
            5. Look at Content Indexing | Queue Contents and see entries not being processed
            6. Shutdown Confluence
            7. Start Confluence and the queue is processed

            If running 5.4 or running 5.8 with the patch, entries are processed in step 5.

            Peter Binney added a comment - Thank you, though I assume I do not deploy the __MACOSX directory and .DS_Store files in the .zip Having deployed the xml and the two .class files, Confluence now starts. Could I suggest that you fix your patch deployment page to cover this. I can confirm that the patch has fixed the problem. For your reference, the steps I use to trigger it are: 1. Run: ALTER DATABASE confluence set single_user with rollback immediate 2. Try to view a page - which hangs or errors as expected 3. Run: ALTER DATABASE confluence set multi_user 4. Edit a page 5. Look at Content Indexing | Queue Contents and see entries not being processed 6. Shutdown Confluence 7. Start Confluence and the queue is processed If running 5.4 or running 5.8 with the patch, entries are processed in step 5.

            Morgan Knicely (Inactive) added a comment -

            Peter,

            Everything in the zip should be placed in the same location relative to the WEB-INF/classes directory.

            For schedulingSubsystemContext.xml this means <installation-directory>/confluence/WEB-INF/classes/ because the file is in the top directory of the zip.

            Cheers!
            -Morgan.

            Morgan Knicely (Inactive) added a comment - Peter, Everything in the zip should be placed in the same location relative to the WEB-INF/classes directory. For schedulingSubsystemContext.xml this means <installation-directory>/confluence/WEB-INF/classes/ because the file is in the top directory of the zip. Cheers! -Morgan.

            Peter Binney added a comment -

            Yes, that is what I followed, but Confluence then fails to start, reporting:

            2016-01-11 09:39:21,314 ERROR [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'managedScheduledJobTriggerFactory' defined in class path resource [schedulingSubsystemContext.xml]
...
2016-01-11 09:39:21,326 ERROR [localhost-startStop-1] [atlassian.config.bootstrap.BootstrappedContextLoaderListener] contextInitialized Failed to initialize the main Confluence context

            which is why I was asking about the .xml file, which presumably also has to be deployed somewhere.

            Peter Binney added a comment - Yes, that is what I followed, but Confluence then fails to start, reporting: 2016-01-11 09:39:21,314 ERROR [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Context initialization failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'managedScheduledJobTriggerFactory' defined in class path resource [schedulingSubsystemContext.xml] ... 2016-01-11 09:39:21,326 ERROR [localhost-startStop-1] [atlassian.config.bootstrap.BootstrappedContextLoaderListener] contextInitialized Failed to initialize the main Confluence context which is why I was asking about the .xml file, which presumably also has to be deployed somewhere.

            James McArthur (Inactive) added a comment -

            peter.binney@commerzbank.com Please follow the instructions linked to in the issue description (https://confluence.atlassian.com/doc/installing-patched-class-files-166528493.html). In those instructions, you will notice it states that any folders which don't exist should be created manually. Cheers!

            James McArthur (Inactive) added a comment - peter.binney@commerzbank.com Please follow the instructions linked to in the issue description ( https://confluence.atlassian.com/doc/installing-patched-class-files-166528493.html ). In those instructions, you will notice it states that any folders which don't exist should be created manually. Cheers!

            Peter Binney added a comment -

            Please will you give proper instructions for applying the patch. The .zip contains two class files:

            com/atlassian/confluence/schedule/listeners/ScheduledJobsListener$1.class
            com/atlassian/confluence/schedule/listeners/ScheduledJobsListener.class

            and the file:

            schedulingSubsystemContext.xml

            The 5.8.18 installation has no classes of these names, no such .xml file, and no folder: <installation-directory>/confluence/WEB-INF/classes/com/atlassian/confluence/schedule

            Peter Binney added a comment - Please will you give proper instructions for applying the patch. The .zip contains two class files: com/atlassian/confluence/schedule/listeners/ScheduledJobsListener$1.class com/atlassian/confluence/schedule/listeners/ScheduledJobsListener.class and the file: schedulingSubsystemContext.xml The 5.8.18 installation has no classes of these names, no such .xml file, and no folder: <installation-directory>/confluence/WEB-INF/classes/com/atlassian/confluence/schedule

              jmcarthur James McArthur (Inactive)
              onevalainen Olli Nevalainen
              Affected customers:
              16 This affects my team
              Watchers:
              49 Start watching this issue

                Created:
                Updated:
                Resolved: