Details
-
Bug
-
Resolution: Timed out
-
Medium
-
30
-
Severity 2 - Major
-
Description
Summary
In Confluence Selecting the Download all option on a page containing a large amount of or just very large attachments, can lead to a page timeout and fail to proceede, and if launched multiple times will lock CPU and crash the system with Timeout waiting for idle object error that will not escape on system restart.
UPDATE:
- This does not seem to be limited to Downloading attachments we have now seen this occur on:
- Download all attachments
- page and Space PDF Export
- Page XML Export
- Viewing large pages
- Viewing page with Content By Label Macro returning large result set
Environment
- Confluence Cloud
Steps to Reproduce
- Go to Confluence Administration >> General Configuration >> EDIT Attachment Settings
- Set Attachment Maximum Size to a large value (EXE: 99999999999999).
- Add multiple Gigs of attachments to a single page in confluence
- Successful test scenario was Five 2+ GB .zip files
- go to page attachments
- the ... dropdown upper Right >> Attachments
- Select Download All
- Page will time out
- Go Back 1 page
- Select Download All a second time
Expected Results
Page should not time out on the first Select Download All operation, and attachments queued for download or error noting that attachments are too large
Actual Results
System CPU pegged at 100% as multiple actions are stuck in <IDLE> in transaction state and fail to complete:
postgres=# SELECT datname, usename, procpid, waiting, query_start, current_query FROM pg_stat_activity order by usename desc; datname | usename | procpid | waiting | query_start | current_query ------------+------------+---------+---------+-------------------------------+------------------------------------------------------------------------ -------------------------------------------- postgres | postgres | 22724 | f | 2015-12-14 21:12:36.18971+00 | SELECT datname, usename, procpid, waiting, query_start, current_query F ROM pg_stat_activity order by usename desc; jira | jira | 22553 | f | 2015-12-14 21:12:22.707282+00 | <IDLE> jira | jira | 22483 | f | 2015-12-14 21:12:04.249722+00 | <IDLE> jira | jira | 22653 | f | 2015-12-14 21:12:22.707079+00 | <IDLE> jira | jira | 22656 | f | 2015-12-14 21:12:22.70747+00 | <IDLE> crowd | crowd | 21903 | f | 2015-12-14 21:12:35.55863+00 | <IDLE> crowd | crowd | 22728 | f | | <IDLE> crowd | crowd | 22651 | f | 2015-12-14 21:11:58.545044+00 | <IDLE> crowd | crowd | 22727 | f | | <IDLE> crowd | crowd | 22626 | f | 2015-12-14 21:12:35.530104+00 | <IDLE> crowd | crowd | 22652 | f | 2015-12-14 21:11:44.278331+00 | <IDLE> crowd | crowd | 22716 | f | 2015-12-14 21:12:35.567079+00 | <IDLE> confluence | confluence | 22594 | f | 2015-12-14 21:03:00.135447+00 | <IDLE> confluence | confluence | 22408 | f | 2015-12-14 21:01:03.962907+00 | <IDLE> in transaction confluence | confluence | 22569 | f | 2015-12-14 21:02:51.301568+00 | <IDLE> in transaction confluence | confluence | 22579 | f | 2015-12-14 21:05:42.383342+00 | <IDLE> confluence | confluence | 22413 | f | 2015-12-14 21:01:03.98642+00 | <IDLE> in transaction confluence | confluence | 22619 | f | 2015-12-14 21:04:00.094216+00 | <IDLE> confluence | confluence | 22438 | f | 2015-12-14 21:01:03.943963+00 | <IDLE> in transaction confluence | confluence | 22517 | f | 2015-12-14 21:01:38.782078+00 | <IDLE> in transaction confluence | confluence | 22460 | f | 2015-12-14 21:01:03.974015+00 | <IDLE> in transaction confluence | confluence | 22567 | f | 2015-12-14 21:03:42.163562+00 | <IDLE> confluence | confluence | 22561 | f | 2015-12-14 21:02:03.783761+00 | <IDLE> in transaction confluence | confluence | 22603 | f | 2015-12-14 21:05:00.24924+00 | <IDLE> confluence | confluence | 22488 | f | 2015-12-14 21:01:38.783423+00 | <IDLE> in transaction confluence | confluence | 22491 | f | 2015-12-14 21:01:03.946036+00 | <IDLE> in transaction confluence | confluence | 22548 | f | 2015-12-14 21:01:50.847113+00 | <IDLE> in transaction confluence | confluence | 22589 | f | 2015-12-14 21:04:42.159784+00 | <IDLE> crowd | collectd | 1032 | f | 2015-12-14 21:12:28.11684+00 | <IDLE> agni | collectd | 1037 | f | 2015-12-14 21:12:21.345539+00 | <IDLE> confluence | collectd | 1034 | f | 2015-12-14 21:12:23.161756+00 | <IDLE> jira | collectd | 1033 | f | 2015-12-14 21:11:44.74128+00 | <IDLE> bamboo | collectd | 1035 | f | 2015-12-14 21:12:21.341521+00 | <IDLE> bamboo | bamboo | 20017 | f | 2015-12-14 21:11:55.024804+00 | <IDLE> bamboo | bamboo | 22293 | f | 2015-12-14 21:12:03.290576+00 | <IDLE> (35 rows)
Error in logs:
2015-12-14 21:38:14.625330500 2015-12-14 21:38:14,623 WARN [http-bio-1990-exec-7] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1450126779117, threadId=180, threadName='http-bio-1990-exec-7', userId='<USER_NAME_HERE>', type='web-request', summary='/pages/downloadallattachments.action?pageId=<PAGE_ID_HERE>'} 2015-12-14 21:38:14.625332500 -- url: /wiki/pages/downloadallattachments.action | traceId: ae0e3ee5d161f917 | userName: <USER_NAME_HERE> | referer: https://<BASE_URL>.atlassian.net/wiki/pages/downloadallattachments.action?pageId=<PAGE_ID_HERE>
Followed by:
2015-12-14 21:40:18.781931500 org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection
2015-12-14 21:40:18.781933500 at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:473)
2015-12-14 21:40:18.781933500 at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374)
2015-12-14 21:40:18.781935500 at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
2015-12-14 21:40:18.781935500 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-12-14 21:40:18.781936500 at java.lang.reflect.Method.invoke(Method.java:497)
2015-12-14 21:40:18.781936500 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
2015-12-14 21:40:18.781938500 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
2015-12-14 21:40:18.781941500 at com.sun.proxy.$Proxy16.getTransaction(Unknown Source)
2015-12-14 21:40:18.781941500 at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263)
Leading into large stack trace ending with
2015-12-14 21:40:18.781968500 Caused by: net.sf.hibernate.exception.GenericJDBCException: Cannot open connection 2015-12-14 21:40:18.781969500 at net.sf.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:80) 2015-12-14 21:40:18.781970500 at net.sf.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:69) 2015-12-14 21:40:18.781970500 at net.sf.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29) 2015-12-14 21:40:18.781973500 at net.sf.hibernate.impl.BatcherImpl.convert(BatcherImpl.java:328) 2015-12-14 21:40:18.781973500 at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:295) 2015-12-14 21:40:18.781974500 at net.sf.hibernate.impl.SessionImpl.connect(SessionImpl.java:3407) 2015-12-14 21:40:18.781974500 at net.sf.hibernate.impl.SessionImpl.connection(SessionImpl.java:3367) 2015-12-14 21:40:18.781974500 at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422) 2015-12-14 21:40:18.781976500 ... 35 more 2015-12-14 21:40:18.781976500 Caused by: org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object 2015-12-14 21:40:18.781977500 at org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114) 2015-12-14 21:40:18.781977500 at org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) 2015-12-14 21:40:18.781979500 at net.sf.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:59) 2015-12-14 21:40:18.781980500 at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:292) 2015-12-14 21:40:18.781980500 ... 38 more 2015-12-14 21:40:18.781980500 Caused by: java.util.NoSuchElementException: Timeout waiting for idle object 2015-12-14 21:40:18.781985500 at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1171) 2015-12-14 21:40:18.781986500 at org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106) 2015-12-14 21:40:18.781986500 ... 41 more
Notes
Restarting Service does nothing as the action resumes on restart, Must Do Container reboot to terminate
Please keep in mind the Atlassian Cloud storage policy and that the system was not designed to be used for file storage, but has light file attachment abilities for collaboration:
Workaround
File a Support Request at https://support.atlassian.com to have service restored Noting a timeframe we can restart the system (All Services will go offline, JIRA, Confluence, Bamboo)
- Download attachments individualy