Loading...

Details

Type: Bug
Resolution: Timed out
Priority: Medium
Component/s: Page - Attachments
Labels:

Support reference count:
30
Symptom Severity:
Severity 2 - Major
Bug Fix Policy:
View Atlassian Cloud bug fix policy

Description

Summary

In Confluence Selecting the Download all option on a page containing a large amount of or just very large attachments, can lead to a page timeout and fail to proceede, and if launched multiple times will lock CPU and crash the system with Timeout waiting for idle object error that will not escape on system restart.

UPDATE:

This does not seem to be limited to Downloading attachments we have now seen this occur on:
- Download all attachments
- page and Space PDF Export
- Page XML Export
- Viewing large pages
- Viewing page with Content By Label Macro returning large result set

Environment

Confluence Cloud

Steps to Reproduce

Go to Confluence Administration >> General Configuration >> EDIT Attachment Settings
1. Set Attachment Maximum Size to a large value (EXE: 99999999999999).
Add multiple Gigs of attachments to a single page in confluence
1. Successful test scenario was Five 2+ GB .zip files
go to page attachments
1. the ... dropdown upper Right >> Attachments
Select Download All
Page will time out
Go Back 1 page
Select Download All a second time

Expected Results

Page should not time out on the first Select Download All operation, and attachments queued for download or error noting that attachments are too large

Actual Results

System CPU pegged at 100% as multiple actions are stuck in <IDLE> in transaction state and fail to complete:

postgres=# SELECT datname, usename, procpid, waiting, query_start, current_query FROM pg_stat_activity order by usename desc;
  datname   |  usename   | procpid | waiting |          query_start          |                                                   current_query        
                                            
------------+------------+---------+---------+-------------------------------+------------------------------------------------------------------------
--------------------------------------------
 postgres   | postgres   |   22724 | f       | 2015-12-14 21:12:36.18971+00  | SELECT datname, usename, procpid, waiting, query_start, current_query F
ROM pg_stat_activity order by usename desc;
 jira       | jira       |   22553 | f       | 2015-12-14 21:12:22.707282+00 | <IDLE>
 jira       | jira       |   22483 | f       | 2015-12-14 21:12:04.249722+00 | <IDLE>
 jira       | jira       |   22653 | f       | 2015-12-14 21:12:22.707079+00 | <IDLE>
 jira       | jira       |   22656 | f       | 2015-12-14 21:12:22.70747+00  | <IDLE>
 crowd      | crowd      |   21903 | f       | 2015-12-14 21:12:35.55863+00  | <IDLE>
 crowd      | crowd      |   22728 | f       |                               | <IDLE>
 crowd      | crowd      |   22651 | f       | 2015-12-14 21:11:58.545044+00 | <IDLE>
 crowd      | crowd      |   22727 | f       |                               | <IDLE>
 crowd      | crowd      |   22626 | f       | 2015-12-14 21:12:35.530104+00 | <IDLE>
 crowd      | crowd      |   22652 | f       | 2015-12-14 21:11:44.278331+00 | <IDLE>
 crowd      | crowd      |   22716 | f       | 2015-12-14 21:12:35.567079+00 | <IDLE>
 confluence | confluence |   22594 | f       | 2015-12-14 21:03:00.135447+00 | <IDLE>
 confluence | confluence |   22408 | f       | 2015-12-14 21:01:03.962907+00 | <IDLE> in transaction
 confluence | confluence |   22569 | f       | 2015-12-14 21:02:51.301568+00 | <IDLE> in transaction
 confluence | confluence |   22579 | f       | 2015-12-14 21:05:42.383342+00 | <IDLE>
 confluence | confluence |   22413 | f       | 2015-12-14 21:01:03.98642+00  | <IDLE> in transaction
 confluence | confluence |   22619 | f       | 2015-12-14 21:04:00.094216+00 | <IDLE>
 confluence | confluence |   22438 | f       | 2015-12-14 21:01:03.943963+00 | <IDLE> in transaction
 confluence | confluence |   22517 | f       | 2015-12-14 21:01:38.782078+00 | <IDLE> in transaction
 confluence | confluence |   22460 | f       | 2015-12-14 21:01:03.974015+00 | <IDLE> in transaction
 confluence | confluence |   22567 | f       | 2015-12-14 21:03:42.163562+00 | <IDLE>
 confluence | confluence |   22561 | f       | 2015-12-14 21:02:03.783761+00 | <IDLE> in transaction
 confluence | confluence |   22603 | f       | 2015-12-14 21:05:00.24924+00  | <IDLE>
 confluence | confluence |   22488 | f       | 2015-12-14 21:01:38.783423+00 | <IDLE> in transaction
 confluence | confluence |   22491 | f       | 2015-12-14 21:01:03.946036+00 | <IDLE> in transaction
 confluence | confluence |   22548 | f       | 2015-12-14 21:01:50.847113+00 | <IDLE> in transaction
 confluence | confluence |   22589 | f       | 2015-12-14 21:04:42.159784+00 | <IDLE>
 crowd      | collectd   |    1032 | f       | 2015-12-14 21:12:28.11684+00  | <IDLE>
 agni       | collectd   |    1037 | f       | 2015-12-14 21:12:21.345539+00 | <IDLE>
 confluence | collectd   |    1034 | f       | 2015-12-14 21:12:23.161756+00 | <IDLE>
 jira       | collectd   |    1033 | f       | 2015-12-14 21:11:44.74128+00  | <IDLE>
 bamboo     | collectd   |    1035 | f       | 2015-12-14 21:12:21.341521+00 | <IDLE>
 bamboo     | bamboo     |   20017 | f       | 2015-12-14 21:11:55.024804+00 | <IDLE>
 bamboo     | bamboo     |   22293 | f       | 2015-12-14 21:12:03.290576+00 | <IDLE>
(35 rows)

Error in logs:

2015-12-14 21:38:14.625330500 2015-12-14 21:38:14,623 WARN [http-bio-1990-exec-7] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1450126779117, threadId=180, threadName='http-bio-1990-exec-7', userId='<USER_NAME_HERE>', type='web-request', summary='/pages/downloadallattachments.action?pageId=<PAGE_ID_HERE>'}
2015-12-14 21:38:14.625332500  -- url: /wiki/pages/downloadallattachments.action | traceId: ae0e3ee5d161f917 | userName: <USER_NAME_HERE> | referer: https://<BASE_URL>.atlassian.net/wiki/pages/downloadallattachments.action?pageId=<PAGE_ID_HERE>

Followed by:

2015-12-14 21:40:18.781931500 org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection
2015-12-14 21:40:18.781933500 	at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:473)
2015-12-14 21:40:18.781933500 	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374)
2015-12-14 21:40:18.781935500 	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
2015-12-14 21:40:18.781935500 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-12-14 21:40:18.781936500 	at java.lang.reflect.Method.invoke(Method.java:497)
2015-12-14 21:40:18.781936500 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
2015-12-14 21:40:18.781938500 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
2015-12-14 21:40:18.781941500 	at com.sun.proxy.$Proxy16.getTransaction(Unknown Source)
2015-12-14 21:40:18.781941500 	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263)

Leading into large stack trace ending with

2015-12-14 21:40:18.781968500 Caused by: net.sf.hibernate.exception.GenericJDBCException: Cannot open connection
2015-12-14 21:40:18.781969500 	at net.sf.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:80)
2015-12-14 21:40:18.781970500 	at net.sf.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:69)
2015-12-14 21:40:18.781970500 	at net.sf.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
2015-12-14 21:40:18.781973500 	at net.sf.hibernate.impl.BatcherImpl.convert(BatcherImpl.java:328)
2015-12-14 21:40:18.781973500 	at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:295)
2015-12-14 21:40:18.781974500 	at net.sf.hibernate.impl.SessionImpl.connect(SessionImpl.java:3407)
2015-12-14 21:40:18.781974500 	at net.sf.hibernate.impl.SessionImpl.connection(SessionImpl.java:3367)
2015-12-14 21:40:18.781974500 	at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422)
2015-12-14 21:40:18.781976500 	... 35 more
2015-12-14 21:40:18.781976500 Caused by: org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
2015-12-14 21:40:18.781977500 	at org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
2015-12-14 21:40:18.781977500 	at org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
2015-12-14 21:40:18.781979500 	at net.sf.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:59)
2015-12-14 21:40:18.781980500 	at net.sf.hibernate.impl.BatcherImpl.openConnection(BatcherImpl.java:292)
2015-12-14 21:40:18.781980500 	... 38 more
2015-12-14 21:40:18.781980500 Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
2015-12-14 21:40:18.781985500 	at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1171)
2015-12-14 21:40:18.781986500 	at org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
2015-12-14 21:40:18.781986500 	... 41 more

Notes

Restarting Service does nothing as the action resumes on restart, Must Do Container reboot to terminate

Please keep in mind the Atlassian Cloud storage policy and that the system was not designed to be used for file storage, but has light file attachment abilities for collaboration:

https://confluence.atlassian.com/cloud/atlassian-cloud-storage-policy-744721619.html

Workaround

File a Support Request at https://support.atlassian.com to have service restored Noting a timeframe we can restart the system (All Services will go offline, JIRA, Confluence, Bamboo)

Download attachments individualy

Large attachment download Crashes System