I've reproduced it locally, although it is not deterministic and may require some patience.
Steps to reproduce
- Set up a fresh installation of Crowd 2.7.0, connected to a database (I'm using Postgres 9.1, although there are customer reporting problems with MySQL and SQL Server as well).
- Once the set up is completed, log in as 'admin'.
- Reload (Cmd-R or similar) the home page repeatedly. Do not wait for it to refresh completely in your browser, just reload as fast as you can.
Observed outcome
Eventually, the page does not refresh anymore. Crowd freezes.
Using the Postgres admin tools, we find that there is a client that is locking the cwd_token table, see attached image.
WORKAROUND
At this point the workaround is to switch the session storage to in-memory from database storage:
Session Configuration
The in-memory storage is explained in more detail on the page linked above.
If you cannot keep crowd up long enough to access the Administration console, or crowd will simply not finish starting up, follow these steps to modify the value directly in the database:
- Shutdown crowd - ensure the pid is stopped
- Perform a database backup
- Connect to the database
- Execute this sql:
update cwd_property set property_value='false' where property_name='database.token.storage.enabled';
- Restart Crowd
- Validate the session storage is now in-memory by Navigating to Administration > Session Config
- is duplicated by
-
CWD-3697 Crowd token storage in database causing Crowd to hang
-
- Closed
-
-
-
- Closed
-
- is related to
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
CWDDEV-151 Loading...
- was split from
-
-
- Closed
-
- was split into
-
-
- Closed
-
- mentioned in
-
-
-
-
-
-
- requirement of
-
(1 was split from, 1 was split into, 6 mentioned in, 1 requirement of)
[CWD-3692] Crowd freezes under heavy load
This issue is not fully resolved. Current efforts being tracked https://jira.atlassian.com/browse/CWD-3768 and https://jira.atlassian.com/browse/CWD-3769
Hi all,
Some customers are still experiencing deadlocks after upgrading to Crowd 2.7.1. We're still investigating these problems as part of a new issue, see CWD-3769. Please watch that issue for updates, and if you're experiencing problems, please comment on that issue indicating your database vendor and version.
Thank you.
Stephen James Perris (Inactive)
added a comment - I tried using jtds-1.2.4 (2.7.0 and 2.7.1 come with 1.2.7) and it didn't help. As previous user suggested I switched to in-memory tokens - so far, so good.
Stephen James Perris (Inactive)
added a comment - I tried using jtds-1.2.4 (2.7.0 and 2.7.1 come with 1.2.7) and it didn't help. As previous user suggested I switched to in-memory tokens - so far, so good. Stephen James Perris (Inactive)
added a comment - I just upgraded to 2.7.1 this morning and I am getting deadlocks on the same table, using sql server 2008. There is a delete from cwd_token that is blocking an insert to cwd_token. I don't know why the delete doesn't just complete - there are only a couple of rows in cwd_token. If i delete the rows manually and restart crowd is appears to come up fine, but a subsequent restart ends up with the same deadlock.
Stephen James Perris (Inactive)
added a comment - I just upgraded to 2.7.1 this morning and I am getting deadlocks on the same table, using sql server 2008. There is a delete from cwd_token that is blocking an insert to cwd_token. I don't know why the delete doesn't just complete - there are only a couple of rows in cwd_token. If i delete the rows manually and restart crowd is appears to come up fine, but a subsequent restart ends up with the same deadlock. 
Simon Graham
added a comment - Can someone please share what the resolution to this ticket was?
adam.brin@asu.edu, switching from database to memory session storage was the only fix that worked for us with 2.7.1.
https://confluence.atlassian.com/display/CROWD/Session+Configuration#SessionConfiguration-token
Simon Graham
added a comment - Can someone please share what the resolution to this ticket was?
adam.brin@asu.edu , switching from database to memory session storage was the only fix that worked for us with 2.7.1.
https://confluence.atlassian.com/display/CROWD/Session+Configuration#SessionConfiguration-token We just got it:
datid | datname | procpid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start | waiting | current_query ---------+---------+---------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 9871885 | crowd | 13258 | 9865061 | crowd | | 129.219.78.171 | | 57160 | 2014-01-22 11:56:09.108331-07 | 2014-01-22 11:56:09.115989-07 | 2014-01-22 11:56:09.116971-07 | t | insert into cwd_token (directory_id, entity_name, random_number, identifier_hash, random_hash, created_date, last_accessed_date, last_accessed_time, duration, id) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) 9871885 | crowd | 12910 | 9865061 | crowd | | 129.219.78.171 | | 56577 | 2014-01-22 11:40:13.587329-07 | 2014-01-22 11:56:08.987633-07 | 2014-01-22 11:56:09.097698-07 | f | <IDLE> in transaction 9871885 | crowd | 13265 | 9865061 | crowd | | 129.219.78.171 | | 57173 | 2014-01-22 11:56:41.07983-07 | 2014-01-22 11:56:41.093217-07 | 2014-01-22 11:56:41.093989-07 | f | <IDLE> in transaction 9871885 | crowd | 13273 | 9865061 | crowd | | 129.219.78.171 | | 57176 | 2014-01-22 11:56:51.473408-07 | | 2014-01-22 11:56:51.486823-07 | f | <IDLE> 9871885 | crowd | 13277 | 9865061 | crowd | | 129.219.78.171 | | 57179 | 2014-01-22 11:57:06.852578-07 | | 2014-01-22 11:57:06.864924-07 | f | <IDLE> 9871885 | crowd | 13295 | 9865061 | crowd | | 129.219.78.171 | | 57183 | 2014-01-22 11:57:31.281612-07 | | 2014-01-22 11:57:31.29432-07 | f | <IDLE> 9871885 | crowd | 13296 | 9865061 | crowd | | 129.219.78.171 | | 57184 | 2014-01-22 11:57:31.297774-07 | | 2014-01-22 11:57:31.943222-07 | f | <IDLE> 9871885 | crowd | 13300 | 9865061 | crowd | | 129.219.78.171 | | 57193 | 2014-01-22 11:57:34.058377-07 | | 2014-01-22 11:57:34.081716-07 | f | <IDLE> 9871885 | crowd | 13301 | 9865061 | crowd | | 129.219.78.171 | | 57194 | 2014-01-22 11:57:34.085876-07 | | 2014-01-22 11:57:35.629233-07 | f | <IDLE> 9871885 | crowd | 13303 | 9865061 | crowd | | 129.219.78.171 | | 57197 | 2014-01-22 11:57:35.633262-07 | | 2014-01-22 11:57:36.927078-07 | f | <IDLE> 9871885 | crowd | 13304 | 9865061 | crowd | | 129.219.78.171 | | 57202 | 2014-01-22 11:57:38.360467-07 | | 2014-01-22 11:57:38.371585-07 | f | <IDLE> 9871885 | crowd | 13322 | 9865061 | crowd | | 129.219.78.171 | | 57212 | 2014-01-22 11:58:18.022795-07 | | 2014-01-22 11:58:23.646191-07 | f | <IDLE> 9871885 | crowd | 13342 | 9865061 | crowd | | 129.219.78.171 | | 57246 | 2014-01-22 11:59:40.56599-07 | | 2014-01-22 11:59:40.587332-07 | f | <IDLE> 9871885 | crowd | 13323 | 9865061 | crowd | | 129.219.78.171 | | 57213 | 2014-01-22 11:58:18.05985-07 | 2014-01-22 11:58:31.954855-07 | 2014-01-22 11:58:31.955814-07 | f | <IDLE> in transaction 9871885 | crowd | 13343 | 9865061 | crowd | | 129.219.78.171 | | 57247 | 2014-01-22 11:59:40.591373-07 | | 2014-01-22 11:59:49.382347-07 | f | <IDLE> 9871885 | crowd | 13348 | 9865061 | crowd | | 129.219.78.171 | | 57250 | 2014-01-22 11:59:59.828893-07 | | 2014-01-22 11:59:59.836861-07 | f | <IDLE> 9871885 | crowd | 13364 | 9865061 | crowd | | 129.219.78.171 | | 57255 | 2014-01-22 12:00:45.362305-07 | 2014-01-22 12:00:45.378749-07 | 2014-01-22 12:00:45.379602-07 | f | <IDLE> in transaction 9871885 | crowd | 13366 | 9865061 | crowd | | 129.219.78.171 | | 57261 | 2014-01-22 12:00:51.342486-07 | | 2014-01-22 12:00:51.353946-07 | f | <IDLE> 9871885 | crowd | 13374 | 9865061 | crowd | | 129.219.78.171 | | 57275 | 2014-01-22 12:01:10.193545-07 | 2014-01-22 12:01:10.207084-07 | 2014-01-22 12:01:10.339138-07 | f | <IDLE> in transaction 9871885 | crowd | 13376 | 9865061 | crowd | | 129.219.78.171 | | 57280 | 2014-01-22 12:01:15.895756-07 | | 2014-01-22 12:01:15.908052-07 | f | <IDLE> 9871885 | crowd | 13379 | 9865061 | crowd | | 129.219.78.171 | | 57285 | 2014-01-22 12:01:29.546374-07 | | 2014-01-22 12:01:29.556863-07 | f | <IDLE> 9871885 | crowd | 13426 | 9865061 | crowd | | 129.219.78.171 | | 57319 | 2014-01-22 12:02:36.186449-07 | | 2014-01-22 12:02:36.197647-07 | f | <IDLE> 9871885 | crowd | 13427 | 9865061 | crowd | | 129.219.78.171 | | 57324 | 2014-01-22 12:02:37.646477-07 | 2014-01-22 12:02:37.657181-07 | 2014-01-22 12:02:37.657975-07 | f | <IDLE> in transaction 9871885 | crowd | 13428 | 9865061 | crowd | | 129.219.78.171 | | 57326 | 2014-01-22 12:02:37.808258-07 | | 2014-01-22 12:02:37.840561-07 | f | <IDLE> 9871885 | crowd | 13694 | 9865061 | crowd | | 129.219.78.171 | | 57341 | 2014-01-22 12:03:17.792378-07 | | 2014-01-22 12:03:18.256047-07 | f | <IDLE> 9871885 | crowd | 13695 | 9865061 | crowd | | 129.219.78.171 | | 57342 | 2014-01-22 12:03:17.801535-07 | | 2014-01-22 12:03:17.823657-07 | f | <IDLE> (26 rows)
currently the query is hanging for 10 minutes. killing query 13258 did not fix it, it required killing 13348 as well:
datid | datname | procpid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start | waiting | current_query ---------+---------+---------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+-------------------------------+------------------------------- +---------+-------------------------------------------------------------------------------------------------- 9871885 | crowd | 13833 | 9865061 | crowd | | 129.219.78.171 | | 57466 | 2014-01-22 12:07:35.792655-07 | | 2014-01-22 12:07:35.809082-07 | f | <IDLE> 9871885 | crowd | 12910 | 9865061 | crowd | | 129.219.78.171 | | 56577 | 2014-01-22 11:40:13.587329-07 | 2014-01-22 11:56:08.987633-07 | 2014-01-22 11:56:09.097698-07 | f | <IDLE> in transaction 9871885 | crowd | 13265 | 9865061 | crowd | | 129.219.78.171 | | 57173 | 2014-01-22 11:56:41.07983-07 | 2014-01-22 12:07:08.688952-07 | 2014-01-22 12:07:08.955126-07 | f | <IDLE> in transaction 9871885 | crowd | 13273 | 9865061 | crowd | | 129.219.78.171 | | 57176 | 2014-01-22 11:56:51.473408-07 | 2014-01-22 12:06:43.497104-07 | 2014-01-22 12:06:43.520063-07 | f | <IDLE> in transaction 9871885 | crowd | 13277 | 9865061 | crowd | | 129.219.78.171 | | 57179 | 2014-01-22 11:57:06.852578-07 | 2014-01-22 12:06:43.491868-07 | 2014-01-22 12:06:43.523918-07 | f | <IDLE> in transaction 9871885 | crowd | 13295 | 9865061 | crowd | | 129.219.78.171 | | 57183 | 2014-01-22 11:57:31.281612-07 | 2014-01-22 12:06:43.498855-07 | 2014-01-22 12:06:43.518806-07 | f | <IDLE> in transaction 9871885 | crowd | 13296 | 9865061 | crowd | | 129.219.78.171 | | 57184 | 2014-01-22 11:57:31.297774-07 | 2014-01-22 12:06:43.500564-07 | 2014-01-22 12:06:43.519405-07 | f | <IDLE> in transaction 9871885 | crowd | 13300 | 9865061 | crowd | | 129.219.78.171 | | 57193 | 2014-01-22 11:57:34.058377-07 | 2014-01-22 12:06:43.502313-07 | 2014-01-22 12:06:43.517582-07 | f | <IDLE> in transaction 9871885 | crowd | 13301 | 9865061 | crowd | | 129.219.78.171 | | 57194 | 2014-01-22 11:57:34.085876-07 | 2014-01-22 12:06:43.503969-07 | 2014-01-22 12:06:43.521057-07 | f | <IDLE> in transaction 9871885 | crowd | 13303 | 9865061 | crowd | | 129.219.78.171 | | 57197 | 2014-01-22 11:57:35.633262-07 | 2014-01-22 12:06:43.493055-07 | 2014-01-22 12:06:43.524514-07 | f | <IDLE> in transaction 9871885 | crowd | 13304 | 9865061 | crowd | | 129.219.78.171 | | 57202 | 2014-01-22 11:57:38.360467-07 | 2014-01-22 12:06:43.505062-07 | 2014-01-22 12:06:43.518187-07 | f | <IDLE> in transaction 9871885 | crowd | 13322 | 9865061 | crowd | | 129.219.78.171 | | 57212 | 2014-01-22 11:58:18.022795-07 | 2014-01-22 12:06:43.506013-07 | 2014-01-22 12:06:43.520428-07 | f | <IDLE> in transaction 9871885 | crowd | 13342 | 9865061 | crowd | | 129.219.78.171 | | 57246 | 2014-01-22 11:59:40.56599-07 | 2014-01-22 12:06:43.506998-07 | 2014-01-22 12:06:43.523104-07 | f | <IDLE> in transaction 9871885 | crowd | 13323 | 9865061 | crowd | | 129.219.78.171 | | 57213 | 2014-01-22 11:58:18.05985-07 | 2014-01-22 11:58:31.954855-07 | 2014-01-22 12:06:43.522368-07 | f | <IDLE> in transaction 9871885 | crowd | 13343 | 9865061 | crowd | | 129.219.78.171 | | 57247 | 2014-01-22 11:59:40.591373-07 | 2014-01-22 12:06:43.494302-07 | 2014-01-22 12:06:43.521782-07 | f | <IDLE> in transaction 9871885 | crowd | 13348 | 9865061 | crowd | | 129.219.78.171 | | 57250 | 2014-01-22 11:59:59.828893-07 | 2014-01-22 12:06:43.52474-07 | 2014-01-22 12:06:43.526993-07 | t | delete from cwd_token where last_accessed_time+$1*1000<$2 or last_accessed_time+duration*1000<$3 9871885 | crowd | 13364 | 9865061 | crowd | | 129.219.78.171 | | 57255 | 2014-01-22 12:00:45.362305-07 | 2014-01-22 12:00:45.378749-07 | 2014-01-22 12:00:45.379602-07 | f | <IDLE> in transaction 9871885 | crowd | 13366 | 9865061 | crowd | | 129.219.78.171 | | 57261 | 2014-01-22 12:00:51.342486-07 | | 2014-01-22 12:00:51.353946-07 | f | <IDLE> 9871885 | crowd | 13374 | 9865061 | crowd | | 129.219.78.171 | | 57275 | 2014-01-22 12:01:10.193545-07 | 2014-01-22 12:01:10.207084-07 | 2014-01-22 12:01:10.339138-07 | f | <IDLE> in transaction 9871885 | crowd | 13376 | 9865061 | crowd | | 129.219.78.171 | | 57280 | 2014-01-22 12:01:15.895756-07 | | 2014-01-22 12:01:15.908052-07 | f | <IDLE> 9871885 | crowd | 13379 | 9865061 | crowd | | 129.219.78.171 | | 57285 | 2014-01-22 12:01:29.546374-07 | | 2014-01-22 12:01:29.556863-07 | f | <IDLE> 9871885 | crowd | 13426 | 9865061 | crowd | | 129.219.78.171 | | 57319 | 2014-01-22 12:02:36.186449-07 | | 2014-01-22 12:02:36.197647-07 | f | <IDLE> 9871885 | crowd | 13427 | 9865061 | crowd | | 129.219.78.171 | | 57324 | 2014-01-22 12:02:37.646477-07 | 2014-01-22 12:02:37.657181-07 | 2014-01-22 12:02:37.657975-07 | f | <IDLE> in transaction 9871885 | crowd | 13428 | 9865061 | crowd | | 129.219.78.171 | | 57326 | 2014-01-22 12:02:37.808258-07 | | 2014-01-22 12:02:37.840561-07 | f | <IDLE> 9871885 | crowd | 13755 | 9865061 | crowd | | 129.219.78.171 | | 57411 | 2014-01-22 12:05:47.218956-07 | | 2014-01-22 12:05:47.223723-07 | f | <IDLE> 9871885 | crowd | 13730 | 9865061 | crowd | | 129.219.78.171 | | 57368 | 2014-01-22 12:04:51.256057-07 | | 2014-01-22 12:04:51.266693-07 | f | <IDLE> 9871885 | crowd | 13759 | 9865061 | crowd | | 129.219.78.171 | | 57424 | 2014-01-22 12:05:56.938532-07 | 2014-01-22 12:05:56.952507-07 | 2014-01-22 12:05:56.953349-07 | f | <IDLE> in transaction 9871885 | crowd | 13772 | 9865061 | crowd | | 129.219.78.171 | | 57433 | 2014-01-22 12:06:21.77371-07 | 2014-01-22 12:06:21.787829-07 | 2014-01-22 12:06:21.788662-07 | f | <IDLE> in transaction 9871885 | crowd | 13777 | 9865061 | crowd | | 129.219.78.171 | | 57439 | 2014-01-22 12:06:41.220769-07 | 2014-01-22 12:06:41.242411-07 | 2014-01-22 12:06:41.250518-07 | f | <IDLE> in transaction 9871885 | crowd | 13779 | 9865061 | crowd | | 129.219.78.171 | | 57441 | 2014-01-22 12:06:41.243921-07 | | 2014-01-22 12:06:41.25373-07
to further clarify – we've just upgraded both the crowd app and rest-client to 2.7.1 and have not yet seen the issue. Can someone please share what the resolution to this ticket was?
We've been seeing this too with Crowd 2.7.0. We believe that this is the query that caused it:
014-01-21 16:56:45,588 ajp-bio-8019-exec-143 WARN [engine.jdbc.spi.SqlExceptionHelper] SQL Error: 0, SQLState: 57014 2014-01-21 16:56:45,589 ajp-bio-8019-exec-143 ERROR [engine.jdbc.spi.SqlExceptionHelper] Batch entry 0 insert into cwd_token (directory_id, entity_name, random_number, identifier_hash, random_hash, created_date, last_accessed_date, last_accessed_time, durati on, id) values ('3276801', 'editor@tdar.org', '6944244361016049004', '6gh8sblisWk1S0tXAMRPyg00', 'cTDegtvfvn1QFTexdJBdPQ00', '2014-01-21 16:43:21.765000 -07:00:00', '2014-01-21 16:43:21.765000 -07:00:00', '1390347801765', NULL, '16515257') was aborted. Call getNextException to see the cause. 2014-01-21 16:56:45,590 ajp-bio-8019-exec-143 WARN [engine.jdbc.spi.SqlExceptionHelper] SQL Error: 0, SQLState: 57014 2014-01-21 16:56:45,590 ajp-bio-8019-exec-143 ERROR [engine.jdbc.spi.SqlExceptionHelper] ERROR: canceling statement due to user request
Upon seeing this in the Postgres 9.1 database we looked and had 20 open connections all with pending transactions. We had configured crowd to use up to 50 database connections, but the Crowd client, and the crowd Rest client were both hung. Killing the query-process in postgres released the queries and crowd returned to normal operations. the majority of the connections were reported by pg_stat_activity as <IDLE>, pending transaction.
Hi simon.graham,
We are sorry to hear that you're still experiencing problems with Crowd 2.7.1. This release contains a number of fixes that should make a difference and improve the stability of the application. Given the complexity of the problem, it is possible that the fixes do not cover all possible cases, or you may be hitting a different bug with similar effects.
We'd like to investigate your case in depth. First we need to gather some data. Please create a support ticket in support.atlassian.com to open a confidential channel to exchange information? Make sure you mention this issue (CWD-3692) in the ticket description to speed up triaging of your case. Feel free to mention me in that ticket so I can immediately access the information as soon as you post it.
Please run the latest release (2.7.1) with database session storage, and attach the following information to your ticket:
- The Crowd support information, available under "Administration > System info > Support information".
- The Crowd logs right after a server crash. First, please make sure to increase the Log4j logging level of package "com.atlassian.crowd" from INFO (default) to DEBUG. You can change this setting under "Administration > Logging & Profiling > Log4j logging".
- A thread dump of Crowd when Crowd has crashed. Follow the instructions here: https://confluence.atlassian.com/display/DOC/Generating+a+Thread+Dump+Externally The instructions are for Confluence, but can be adapted for Crowd.
- A report of the state of the database connections right after the crash. You can get this from the pg_stat_activity table in Postgres, see the database documentation for more details (http://www.postgresql.org/docs/8.4/static/monitoring-stats.html).
Thank you.
dberrueta The problem exists even with Crowd >=3.1.1, we have the problem with one of our clients recently and has caused a production downtime of 1.5h.