-
Bug
-
Resolution: Fixed
-
High
-
5.5.6, 5.6.6, 5.8.4, 5.8.5, 5.8.9, 5.9.1, 5.8.10, 5.8.14, 5.9.2, 5.8.17, 5.8.18
-
None
Database request is performed before each schedule job is run. If this request fails for any reason (e.g. connection timeout, not enough DB connections in the pool, table missing in DB), the scheduled job will be prevented from running until the instance is restarted.
This is the relevant stack trace:
org.quartz.SchedulerException: JobListener 'ScheduledJobListener' threw exception: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection [See nested exception: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection] at org.quartz.core.QuartzScheduler.notifyJobListenersToBeExecuted(QuartzScheduler.java:1951) at org.quartz.core.JobRunShell.notifyListenersBeginning(JobRunShell.java:364) at org.quartz.core.JobRunShell.run(JobRunShell.java:190) at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool.lambda$runInThread$187(ConfluenceQuartzThreadPool.java:19) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549) Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:473) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374) at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198) at com.sun.proxy.$Proxy33.getTransaction(Unknown Source) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy142.jobToBeExecuted(Unknown Source) at org.quartz.core.QuartzScheduler.notifyJobListenersToBeExecuted(QuartzScheduler.java:1949) ... 4 more Caused by: net.sf.hibernate.exception.GenericJDBCException: Cannot open connection at net.sf.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:80) at net.sf.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:69) at net.sf.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29) at net.sf.hibernate.impl.BatcherImpl.convert(BatcherImpl.java:328) at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:295) at net.sf.hibernate.impl.SessionImpl.connection(SessionImpl.java:3367) at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422) ... 17 more Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:126) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412) at net.sf.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:59) at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:292) ... 20 more at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446) at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360) at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118) ... 23 more
Two job listeners are called when execution is starting:
- ExecutingJobsManager keeps a list of jobs which are executing
- ScheduledJobsListener saves execution details to database
If ScheduledJobsListener throws an exception, ExecutingJobsManager will not get notified that the execution got cancelled. Further executions get cancelled because AbstractJob#executeInternal(..) checks ExecutingJobsManager to see if similar (but not same) execution is already happening.
Workaround for Confluence 5.8
For users of Confluence 5.8.18, we are providing a patch file
containing a fix for this issue. Please be aware that this patch has not been fully tested and is applied at your own risk. We strongly recommend that you test the patch on a staging instance of Confluence before applying it to a production instance. Refer to https://confluence.atlassian.com/doc/installing-patched-class-files-166528493.html for details on how to apply the patch. Also, make sure that the MaxWaitMillis is set to 10000 so that the connection pool got sometime to recover.
- relates to
-
-
- Closed
-
- was split from
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
BSERV-3169 Able to send HTML email
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
- caused
-
HOT-35329 Loading...
- causes
-
- included in
-
![[JIRA]](/images/icons/generic_link_16.png "[JIRA]")
CPU-171
Confluence 6.0.0-OD-2015.52.1-0001
-
CPU-179
Confluence 6.0.0-OD-2016.01.1-0002
-
CPU-180
Confluence 6.0.0-OD-2016.01.1-0003
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
[CONFSERVER-39825] Quartz scheduled jobs can stop running until restart
It looks like you might be seeing a different issue. If you could raise a support case with us at https://support.atlassian.com/contact/, a support engineer will be able to help you with this.
Thanks,
David Mason
Hello there! Does anybody still watching this issue?
I did as you all said, but I have 500 error
My confluence version 5.9.11
Dmitry Bod
added a comment - - edited Hello there! Does anybody still watching this issue?
I did as you all said, but I have 500 error
My confluence version 5.9.11 Hi vbrimmer,
Yes we don't need to patch it when upgrading to 6.0.3
Thanks,
Minh Tran
Confluence BugMaster
So I patched 5.8.x and now am upgrading that instance to 6.0.3. Is it safe to assume that I do not have to go back in and re-add any of the files that were included in the patch .zip? They show as "added files" in the "5.8.13 modifications" file that the wizard produced.
Peter Binney
added a comment - Thank you, though I assume I do not deploy the __MACOSX directory and .DS_Store files in the .zip
Having deployed the xml and the two .class files, Confluence now starts.
Could I suggest that you fix your patch deployment page to cover this.
I can confirm that the patch has fixed the problem. For your reference, the steps I use to trigger it are:
1. Run: ALTER DATABASE confluence set single_user with rollback immediate
2. Try to view a page - which hangs or errors as expected
3. Run: ALTER DATABASE confluence set multi_user
4. Edit a page
5. Look at Content Indexing | Queue Contents and see entries not being processed
6. Shutdown Confluence
7. Start Confluence and the queue is processed
If running 5.4 or running 5.8 with the patch, entries are processed in step 5.
Peter Binney
added a comment - Thank you, though I assume I do not deploy the __MACOSX directory and .DS_Store files in the .zip
Having deployed the xml and the two .class files, Confluence now starts.
Could I suggest that you fix your patch deployment page to cover this.
I can confirm that the patch has fixed the problem. For your reference, the steps I use to trigger it are:
1. Run: ALTER DATABASE confluence set single_user with rollback immediate
2. Try to view a page - which hangs or errors as expected
3. Run: ALTER DATABASE confluence set multi_user
4. Edit a page
5. Look at Content Indexing | Queue Contents and see entries not being processed
6. Shutdown Confluence
7. Start Confluence and the queue is processed
If running 5.4 or running 5.8 with the patch, entries are processed in step 5. Peter,
Everything in the zip should be placed in the same location relative to the WEB-INF/classes directory.
For schedulingSubsystemContext.xml this means <installation-directory>/confluence/WEB-INF/classes/ because the file is in the top directory of the zip.
Cheers!
-Morgan.
Peter Binney
added a comment - Yes, that is what I followed, but Confluence then fails to start, reporting:
2016-01-11 09:39:21,314 ERROR [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Context initialization failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'managedScheduledJobTriggerFactory' defined in class path resource [schedulingSubsystemContext.xml] ... 2016-01-11 09:39:21,326 ERROR [localhost-startStop-1] [atlassian.config.bootstrap.BootstrappedContextLoaderListener] contextInitialized Failed to initialize the main Confluence context
which is why I was asking about the .xml file, which presumably also has to be deployed somewhere.
Peter Binney
added a comment - Yes, that is what I followed, but Confluence then fails to start, reporting:
2016-01-11 09:39:21,314 ERROR [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'managedScheduledJobTriggerFactory' defined in class path resource [schedulingSubsystemContext.xml]
...
2016-01-11 09:39:21,326 ERROR [localhost-startStop-1] [atlassian.config.bootstrap.BootstrappedContextLoaderListener] contextInitialized Failed to initialize the main Confluence context
which is why I was asking about the .xml file, which presumably also has to be deployed somewhere.
peter.binney@commerzbank.com Please follow the instructions linked to in the issue description (https://confluence.atlassian.com/doc/installing-patched-class-files-166528493.html). In those instructions, you will notice it states that any folders which don't exist should be created manually. Cheers!
Peter Binney
added a comment - Please will you give proper instructions for applying the patch. The .zip contains two class files:
com/atlassian/confluence/schedule/listeners/ScheduledJobsListener$1.class
com/atlassian/confluence/schedule/listeners/ScheduledJobsListener.class
and the file:
schedulingSubsystemContext.xml
The 5.8.18 installation has no classes of these names, no such .xml file, and no folder: <installation-directory>/confluence/WEB-INF/classes/com/atlassian/confluence/schedule
Peter Binney
added a comment - Please will you give proper instructions for applying the patch. The .zip contains two class files:
com/atlassian/confluence/schedule/listeners/ScheduledJobsListener$1.class
com/atlassian/confluence/schedule/listeners/ScheduledJobsListener.class
and the file:
schedulingSubsystemContext.xml
The 5.8.18 installation has no classes of these names, no such .xml file, and no folder: <installation-directory>/confluence/WEB-INF/classes/com/atlassian/confluence/schedule This issue is mentioned in commits included in the Confluence 6.0.0-OD-2016.01.1-0003 release being approved for production: CPU-180
This issue is mentioned in commits included in the Confluence 6.0.0-OD-2016.01.1-0002 release being approved for production: CPU-179
Dear All,
Thanks for raising your concerns. We decided to create a patch for Confluence version 5.8.18
The information for the patch is put in the ticket description
Please let us know if you need any kind of help
Best regards,
Minh Tran
Confluence BugMaster
Atlassian
We are are similarly annoyed. We started the (non-trivial) process to upgrade from 5.4 to 5.8 in June. In September, shortly before releasing to production, we hit this problem. After a couple of months grasping at straws, Atlassian support finally connected the problem to this bug.
But, we are a SQL Server 2008R2 shop (just starting to move to 2014!) and 5.9 no longer supports 2008. So unless Atlassian provide a fix for 5.8 we will be unable to upgrade before 5.4 end-of-life in Mar-2016.
Peter Binney
added a comment - - edited We are are similarly annoyed. We started the (non-trivial) process to upgrade from 5.4 to 5.8 in June. In September, shortly before releasing to production, we hit this problem. After a couple of months grasping at straws, Atlassian support finally connected the problem to this bug.
But, we are a SQL Server 2008R2 shop (just starting to move to 2014!) and 5.9 no longer supports 2008. So unless Atlassian provide a fix for 5.8 we will be unable to upgrade before 5.4 end-of-life in Mar-2016. Thanks Rosario. This is exactly what I feel.
I’m new in managing our Confluence platform, and I have no IT background. But what I see is that keeping the pace that is required here would not work for my organization. We recently upgraded from 5.2.5 to 5,8,14 because it reached “end of life” in Sept. I thought the platform would work for us for some time before we had to upgrade again. Each time, we lose features, we gain bugs and users are definitively affected at the end. At my end, we need a working and a stable wiki. We expect that bugs are fixed and for all versions affected. If we can’t get the proper level of maintenance, then this is not a good business model for us!
@James McArthur
Dear James,
We bought a tool with some requirements and pay for maintenance and we expect the issues in the tool to be fixed as long as we pay for maintenance!
Version 5.9 still supports the Office Macro, but this feature and the Documentation layout (2 features that we use extensively) will be dropped and are no longer supported, therefore to upgrade means to have to compromise between a working product and a product that responds to the requirements it was bought for...
I know that there are alternatives to the Office macros but the only one that seems to work in an equivalent way is not for free.
This is not acceptable: we expect that maintenance (i.e. fixes) for which we pay do not result in a reduction of the feature set or in extra costs!
Please, treat this as an official complaint of a paying Customer!
Regards,
Rosario
This issue is mentioned in release Confluence 6.0.0-OD-2015.52.1-0001 just promoted to jirastudio-prd-virtual
Can you please tell me the release dates for 5.9.3 and 6.0
rosario.dedomenico peter.binney@commerzbank.com eveline.landa1749338496 Unfortunately there will not be any more 5.8 bugfix releases, so the only way to resolve this issue is to upgrade to 5.9.
Peter Binney
added a comment - ditto What about 5.8? We are not willing to move to 5.9 because of the removal of support for essential features.
This issue is mentioned in commits included in the Confluence 6.0.0-OD-2015.52.1-0001 release being approved for production: CPU-171
Content Indexing stopped working correctly in our 5.8.6. production instance Nov 28th 2015.
I have version 5.8.14 and I also encountered this issue. I would need a fix within 5.8 because I might not be available to upgrade to 5.9 given the changes in Documentation layout and removal of Office macros.
The fix has been in master since around last Thursday.
Note:
- The fix is only for jobs based on atlassian-scheduler. Jobs inside Confluence core (indexing jobs etc.) have been converted to be based on atlassian-scheduler.
- Jobs based on Quartz (jobs outside Confluence core) still have problem mentioned in this issue.
Peter Binney
added a comment - This is not a 5.9 only problem. As per https://jira.atlassian.com/browse/CONF-40064 it was introduced in 5.6 and prevents us upgrading to 5.8 (ref: CSP-157392)
We cannot upgrade to 5.9 because of significant requirement changes.
Peter Binney
added a comment - This is not a 5.9 only problem. As per https://jira.atlassian.com/browse/CONF-40064 it was introduced in 5.6 and prevents us upgrading to 5.8 (ref: CSP-157392)
We cannot upgrade to 5.9 because of significant requirement changes. If there's no problem, the branch containing the fix will be merged to master again (it was reverted) in around a day or two.
Will this bug be fixed in the next release? It is causing lots of problems on our side, at the moment as a workaround we have to restart confluence each morning.
Yes. Internal jobs in Confluence core, like the indexing job, have been migrated to be based on Caesium. Thus the problem you mentioned have been fixed for internal jobs.
At first, Quartz will only be deprecated, it won't be removed immediately. ScheduledJobsListener above will still be used. (Outside) jobs that haven't been migrated still have this problem.
ndao - I'm I right in presuming that this code will be replaced soon as we will switch to Caesium?
Hello David!
I have already fixed it today but I did create an issue!
By the way, here it is https://getsupport.atlassian.com/servicedesk/customer/portal/14/CSP-200534
Best wishes!