-
Bug
-
Resolution: Fixed
-
High
-
8.2.0, 8.4.0, 8.5.0, 8.2.1, 8.2.3, 8.3.2, 8.5.1
-
None
-
8
-
Severity 2 - Major
-
82
-
Issue Summary
This issue is related to the removal of user group membership during user directory synchronization. It is reproducible with Jira User Directory, Crowd User Directory, and AD. This issue has been present since version 8.2.0 onwards. I have tested up to version 8.1.4, and the issue could not be reproduced.
According to this issue, if Confluence receives more than one user's remove group request, the Confluence Full directory sync throws the following error, however incremental sync works properly. In the log snippet below, the Hibernate5BatchProcessor is processing two user records in one batch. Initially, this process opens a session and then begins the transaction. However, it fails with the error "Illegally attempted to associate a proxy with two open sessions".
2023-07-09 20:15:43,583 DEBUG [Caesium-1-4] [crowd.hibernate.batch.Hibernate5BatchProcessor] execute processing [ 2 ] objects in [ 1 ] batches of [ 100 ] with [ com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation ]
2023-07-09 20:15:43,583 DEBUG [Caesium-1-4] [crowd.hibernate.batch.Hibernate5BatchProcessor] openSession using session [ SessionImpl(1522555148<open>) ]
2023-07-09 20:15:43,583 DEBUG [Caesium-1-4] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@74303143 ]
2023-07-09 20:15:43,587 ERROR [Caesium-1-4] [org.hibernate.proxy.AbstractLazyInitializer] setSession HHH000485: Illegally attempted to associate a proxy for entity [com.atlassian.crowd.model.user.InternalUser] with id [491524] with two open sessions.
When Confluence encounters this failed batch processing, it switches to individual processing.
2023-07-09 20:15:43,592 WARN [Caesium-1-4] [crowd.hibernate.batch.Hibernate5BatchProcessor] processBatch batch failed falling back to individual processing javax.persistence.PersistenceException: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491524] with two open Sessions at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:188) at org.hibernate.internal.SessionImpl.fireDelete(SessionImpl.java:950) at org.hibernate.internal.SessionImpl.delete(SessionImpl.java:872) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:15) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:11) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.performOperation(AbstractBatchProcessor.java:217) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processBatch(AbstractBatchProcessor.java:156) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:133) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:82) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeAll(HibernateMembershipDao.java:503) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeUsersFromGroup(HibernateMembershipDao.java:386) at com.atlassian.confluence.impl.user.crowd.CachedCrowdMembershipDao.removeUsersFromGroup(CachedCrowdMembershipDao.java:146) .... Caused by: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491524] with two open Sessions at org.hibernate.proxy.AbstractLazyInitializer.setSession(AbstractLazyInitializer.java:134)
Now, during the individual processing, Confluence starts processing the first record by beginning the transaction, committing the transaction, flushing the session, and clearing the session. However, after processing the first record, it once again encounters the same error as before for the remaining user records.
First Record Processing
2023-07-09 19:50:18,614 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual processing [ 2 ] individually
2023-07-09 19:50:18,614 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual processed [ HibernateMembership{groupMember=null, id=589833, parentGroup=com.atlassian.crowd.model.group.InternalGroup@409ee6f2[id=425989,name=test-group,type=GROUP,active=true,description=<null>,lowerName=test-group,createdDate=2023-07-09 19:46:46.71,updatedDate=2023-07-09 19:46:46.71,directoryId=360450,externalId=<null>], userMember=com.atlassian.crowd.model.user.InternalUser@3e045818[id=491524,name=testuser2,createdDate=Sun Jul 09 19:46:46 IST 2023,updatedDate=Sun Jul 09 19:46:46 IST 2023,active=true,emailAddress=testuser2@test.com,firstName=,lastName=testuser2,displayName=testuser2,credential=com.atlassian.crowd.embedded.api.PasswordCredential@783baecd[credential=********,encryptedCredential=true],lowerName=testuser2,lowerEmailAddress=testuser2@test.com,lowerFirstName=,lowerLastName=testuser2,lowerDisplayName=testuser2,directoryId=360450,externalId=1:f34de7f2-88d1-4fcd-a2e5-39323efd4819]} ] [ 50.0% ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] commitTransaction commit transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] flushSession flush session [ SessionImpl(1974468763<open>) ]
2023-07-09 19:50:18,616 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] clearSession clear session [ SessionImpl(1974468763<open>) ]
Second record processing
2023-07-09 19:50:18,617 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ] 2023-07-09 19:50:18,617 ERROR [Caesium-1-1] [org.hibernate.proxy.AbstractLazyInitializer] setSession HHH000485: Illegally attempted to associate a proxy for entity [com.atlassian.crowd.model.user.InternalUser] with id [491526] with two open sessions. -- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440 2023-07-09 19:50:18,617 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] clearSession clear session [ SessionImpl(1974468763<open>) ] 2023-07-09 19:50:18,617 ERROR [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual Could not process class com.atlassian.crowd.embedded.hibernate2.HibernateMembership: HibernateMembership{groupMember=null, id=589834, parentGroup=com.atlassian.crowd.model.group.InternalGroup@409ee6f2[id=425989,name=test-group,type=GROUP,active=true,description=<null>,lowerName=test-group,createdDate=2023-07-09 19:46:46.71,updatedDate=2023-07-09 19:46:46.71,directoryId=360450,externalId=<null>], userMember=com.atlassian.crowd.model.user.InternalUser@1fbf093e[id=491526,name=testuser1,createdDate=Sun Jul 09 19:46:46 IST 2023,updatedDate=Sun Jul 09 19:46:46 IST 2023,active=true,emailAddress=testuser1@test.com,firstName=,lastName=testuser1,displayName=testuser1,credential=com.atlassian.crowd.embedded.api.PasswordCredential@4305f260[credential=********,encryptedCredential=true],lowerName=testuser1,lowerEmailAddress=testuser1@test.com,lowerFirstName=,lowerLastName=testuser1,lowerDisplayName=testuser1,directoryId=360450,externalId=1:15f4409c-4abf-408a-9783-7ede09452550]} -- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440 javax.persistence.PersistenceException: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491526] with two open Sessions at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:188) at org.hibernate.internal.SessionImpl.fireDelete(SessionImpl.java:950) at org.hibernate.internal.SessionImpl.delete(SessionImpl.java:872) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:15) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:11) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.performOperation(AbstractBatchProcessor.java:217) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processIndividual(AbstractBatchProcessor.java:188) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processBatch(AbstractBatchProcessor.java:172) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:133) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:82) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeAll(HibernateMembershipDao.java:503) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeUsersFromGroup(HibernateMembershipDao.java:386) at com.atlassian.confluence.impl.user.crowd.CachedCrowdMembershipDao.removeUsersFromGroup(CachedCrowdMembershipDao.java:146) .... Caused by: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491526] with two open Sessions at org.hibernate.proxy.AbstractLazyInitializer.setSession(AbstractLazyInitializer.java:134)
And this cycle continues until all the records under the batch are processed. During this process, it is observed that during individual processing, Confluence only processes the first record from the batch, while the rest encounter the same "illegally attempted to associate proxy with two open Sessions" error.
However, despite the errors, Confluence marks this directory sync as completed.
This is reproducible on Data Center: (yes).
Note
Same issue also happens when: * Setup LDAP with nested group
- Remove multiple nested groups from parent group then synchronise
As LDAP is always using full sync which will process remove operation in batch
Steps to Reproduce
- Step 1: Create a Jira User Directory in Confluence.
- Step 2: In Jira, create two or more users and assign them to a group, let's call it "test_group" in Jira.
- Step 3: Enable the following debug packages in Confluence as TRACE
com.atlassian.confluence.impl.user.crowd.hibernate com.atlassian.confluence.impl.user.crowd.hibernate.batch
- Step 4: Sync this User Directory with Confluence. During this sync, Confluence processes all the records and creates corresponding users in Confluence.
- Step 5: Remove the "test_group" group from the two users that were created earlier in Jira.
- Step 6: Trigger the Jira User Directory sync again in Confluence, and you will encounter the same error mentioned earlier.
- Step 7: During each sync, Confluence removes one user from the group in Confluence as part of the individual processing.
Expected Results
The process of removing group membership for multiple users in a batch should be executed without encountering the error "Illegally attempted to associate a proxy with two open sessions." If the batch fails due to any other reason, the individual processing will still proceed and process all the records within the same batch without encountering any failure.
Actual Results
During user directory synchronization, the removal of user group membership encounters a failure in the batch operation "com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation". This failure is accompanied by the error message "Illegally attempted to associate a proxy with two open sessions."
2023-07-09 19:50:18,601 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] execute processing [ 2 ] objects in [ 1 ] batches of [ 100 ] with [ com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation ] 2023-07-09 19:50:18,601 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] openSession using session [ SessionImpl(1974468763<open>) ] 2023-07-09 19:50:18,601 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ] 2023-07-09 19:50:18,608 ERROR [Caesium-1-1] [org.hibernate.proxy.AbstractLazyInitializer] setSession HHH000485: Illegally attempted to associate a proxy for entity [com.atlassian.crowd.model.user.InternalUser] with id [491524] with two open sessions. -- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440 2023-07-09 19:50:18,613 WARN [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processBatch batch failed falling back to individual processing -- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440 javax.persistence.PersistenceException: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491524] with two open Sessions at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:188) at org.hibernate.internal.SessionImpl.fireDelete(SessionImpl.java:950) at org.hibernate.internal.SessionImpl.delete(SessionImpl.java:872) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:15) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:11) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.performOperation(AbstractBatchProcessor.java:217) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processBatch(AbstractBatchProcessor.java:156) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:133) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:82) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeAll(HibernateMembershipDao.java:503) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeUsersFromGroup(HibernateMembershipDao.java:386) at com.atlassian.confluence.impl.user.crowd.CachedCrowdMembershipDao.removeUsersFromGroup(CachedCrowdMembershipDao.java:146)
Due to the aforementioned error, the system switches to individual processing. However, during the individual processing, only the first record is successfully processed, while the rest encounter the same error message.
2023-07-09 19:50:18,614 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual processing [ 2 ] individually
2023-07-09 19:50:18,614 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual processed [ HibernateMembership{groupMember=null, id=589833, parentGroup=com.atlassian.crowd.model.group.InternalGroup@409ee6f2[id=425989,name=test-group,type=GROUP,active=true,description=<null>,lowerName=test-group,createdDate=2023-07-09 19:46:46.71,updatedDate=2023-07-09 19:46:46.71,directoryId=360450,externalId=<null>], userMember=com.atlassian.crowd.model.user.InternalUser@3e045818[id=491524,name=testuser2,createdDate=Sun Jul 09 19:46:46 IST 2023,updatedDate=Sun Jul 09 19:46:46 IST 2023,active=true,emailAddress=testuser2@test.com,firstName=,lastName=testuser2,displayName=testuser2,credential=com.atlassian.crowd.embedded.api.PasswordCredential@783baecd[credential=********,encryptedCredential=true],lowerName=testuser2,lowerEmailAddress=testuser2@test.com,lowerFirstName=,lowerLastName=testuser2,lowerDisplayName=testuser2,directoryId=360450,externalId=1:f34de7f2-88d1-4fcd-a2e5-39323efd4819]} ] [ 50.0% ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] commitTransaction commit transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] flushSession flush session [ SessionImpl(1974468763<open>) ]
2023-07-09 19:50:18,616 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] clearSession clear session [ SessionImpl(1974468763<open>) ]
2023-07-09 19:50:18,617 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,617 ERROR [Caesium-1-1] [org.hibernate.proxy.AbstractLazyInitializer] setSession HHH000485: Illegally attempted to associate a proxy for entity [com.atlassian.crowd.model.user.InternalUser] with id [491526] with two open sessions.
-- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440
2023-07-09 19:50:18,617 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] clearSession clear session [ SessionImpl(1974468763<open>) ]
2023-07-09 19:50:18,617 ERROR [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual Could not process class com.atlassian.crowd.embedded.hibernate2.HibernateMembership: HibernateMembership{groupMember=null, id=589834, parentGroup=com.atlassian.crowd.model.group.InternalGroup@409ee6f2[id=425989,name=test-group,type=GROUP,active=true,description=<null>,lowerName=test-group,createdDate=2023-07-09 19:46:46.71,updatedDate=2023-07-09 19:46:46.71,directoryId=360450,externalId=<null>], userMember=com.atlassian.crowd.model.user.InternalUser@1fbf093e[id=491526,name=testuser1,createdDate=Sun Jul 09 19:46:46 IST 2023,updatedDate=Sun Jul 09 19:46:46 IST 2023,active=true,emailAddress=testuser1@test.com,firstName=,lastName=testuser1,displayName=testuser1,credential=com.atlassian.crowd.embedded.api.PasswordCredential@4305f260[credential=********,encryptedCredential=true],lowerName=testuser1,lowerEmailAddress=testuser1@test.com,lowerFirstName=,lowerLastName=testuser1,lowerDisplayName=testuser1,directoryId=360450,externalId=1:15f4409c-4abf-408a-9783-7ede09452550]}
-- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440
javax.persistence.PersistenceException: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491526] with two open Sessions
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:188)
at org.hibernate.internal.SessionImpl.fireDelete(SessionImpl.java:950)
at org.hibernate.internal.SessionImpl.delete(SessionImpl.java:872)
at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:15)
at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:11)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.performOperation(AbstractBatchProcessor.java:217)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processIndividual(AbstractBatchProcessor.java:188)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processBatch(AbstractBatchProcessor.java:172)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:133)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:82)
at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeAll(HibernateMembershipDao.java:503)
at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeUsersFromGroup(HibernateMembershipDao.java:386)
at com.atlassian.confluence.impl.user.crowd.CachedCrowdMembershipDao.removeUsersFromGroup(CachedCrowdMembershipDao.java:146)
Workaround
- This issue is not reproduced while performing the Increment user directory sync.
- Or we can remove the group membership for users who have changes in the source system such as Jira, Crowd, or AD, execute the following SQL query to fetch the relevant records from the cwd_membership table. Make sure to update the Directory_ID, Group Name, and user lower user name accordingly:
select hibernatem0_.id as id1_32_, hibernatem0_.parent_id as parent_i2_32_, hibernatem0_.child_group_id as child_gr3_32_, hibernatem0_.child_user_id as child_us4_32_ from cwd_membership hibernatem0_ cross join cwd_group internalgr1_ cross join cwd_user internalus3_ where hibernatem0_.parent_id = internalgr1_.id and hibernatem0_.child_user_id = internalus3_.id and internalgr1_.directory_id = <Directory_ID> and internalgr1_.lower_group_name = <Group_Name> and (internalus3_.lower_user_name in (? , ?));
Once we have those records, we can run the simple delete statement:
delete from cwd_membership where id=?
- mentioned in
-
Page Loading...
Performing User Directory Full Sync from Jira, Crowd, AD does not remove group membership for multiple users/ nested sub groups in a batch
-
-
Resolution: Fixed
-
High
-
8.2.0, 8.4.0, 8.5.0, 8.2.1, 8.2.3, 8.3.2, 8.5.1
-
None
-
8
-
Severity 2 - Major
-
82
-
Issue Summary
This issue is related to the removal of user group membership during user directory synchronization. It is reproducible with Jira User Directory, Crowd User Directory, and AD. This issue has been present since version 8.2.0 onwards. I have tested up to version 8.1.4, and the issue could not be reproduced.
According to this issue, if Confluence receives more than one user's remove group request, the Confluence Full directory sync throws the following error, however incremental sync works properly. In the log snippet below, the Hibernate5BatchProcessor is processing two user records in one batch. Initially, this process opens a session and then begins the transaction. However, it fails with the error "Illegally attempted to associate a proxy with two open sessions".
2023-07-09 20:15:43,583 DEBUG [Caesium-1-4] [crowd.hibernate.batch.Hibernate5BatchProcessor] execute processing [ 2 ] objects in [ 1 ] batches of [ 100 ] with [ com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation ]
2023-07-09 20:15:43,583 DEBUG [Caesium-1-4] [crowd.hibernate.batch.Hibernate5BatchProcessor] openSession using session [ SessionImpl(1522555148<open>) ]
2023-07-09 20:15:43,583 DEBUG [Caesium-1-4] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@74303143 ]
2023-07-09 20:15:43,587 ERROR [Caesium-1-4] [org.hibernate.proxy.AbstractLazyInitializer] setSession HHH000485: Illegally attempted to associate a proxy for entity [com.atlassian.crowd.model.user.InternalUser] with id [491524] with two open sessions.
When Confluence encounters this failed batch processing, it switches to individual processing.
2023-07-09 20:15:43,592 WARN [Caesium-1-4] [crowd.hibernate.batch.Hibernate5BatchProcessor] processBatch batch failed falling back to individual processing javax.persistence.PersistenceException: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491524] with two open Sessions at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:188) at org.hibernate.internal.SessionImpl.fireDelete(SessionImpl.java:950) at org.hibernate.internal.SessionImpl.delete(SessionImpl.java:872) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:15) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:11) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.performOperation(AbstractBatchProcessor.java:217) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processBatch(AbstractBatchProcessor.java:156) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:133) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:82) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeAll(HibernateMembershipDao.java:503) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeUsersFromGroup(HibernateMembershipDao.java:386) at com.atlassian.confluence.impl.user.crowd.CachedCrowdMembershipDao.removeUsersFromGroup(CachedCrowdMembershipDao.java:146) .... Caused by: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491524] with two open Sessions at org.hibernate.proxy.AbstractLazyInitializer.setSession(AbstractLazyInitializer.java:134)
Now, during the individual processing, Confluence starts processing the first record by beginning the transaction, committing the transaction, flushing the session, and clearing the session. However, after processing the first record, it once again encounters the same error as before for the remaining user records.
First Record Processing
2023-07-09 19:50:18,614 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual processing [ 2 ] individually
2023-07-09 19:50:18,614 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual processed [ HibernateMembership{groupMember=null, id=589833, parentGroup=com.atlassian.crowd.model.group.InternalGroup@409ee6f2[id=425989,name=test-group,type=GROUP,active=true,description=<null>,lowerName=test-group,createdDate=2023-07-09 19:46:46.71,updatedDate=2023-07-09 19:46:46.71,directoryId=360450,externalId=<null>], userMember=com.atlassian.crowd.model.user.InternalUser@3e045818[id=491524,name=testuser2,createdDate=Sun Jul 09 19:46:46 IST 2023,updatedDate=Sun Jul 09 19:46:46 IST 2023,active=true,emailAddress=testuser2@test.com,firstName=,lastName=testuser2,displayName=testuser2,credential=com.atlassian.crowd.embedded.api.PasswordCredential@783baecd[credential=********,encryptedCredential=true],lowerName=testuser2,lowerEmailAddress=testuser2@test.com,lowerFirstName=,lowerLastName=testuser2,lowerDisplayName=testuser2,directoryId=360450,externalId=1:f34de7f2-88d1-4fcd-a2e5-39323efd4819]} ] [ 50.0% ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] commitTransaction commit transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] flushSession flush session [ SessionImpl(1974468763<open>) ]
2023-07-09 19:50:18,616 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] clearSession clear session [ SessionImpl(1974468763<open>) ]
Second record processing
2023-07-09 19:50:18,617 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ] 2023-07-09 19:50:18,617 ERROR [Caesium-1-1] [org.hibernate.proxy.AbstractLazyInitializer] setSession HHH000485: Illegally attempted to associate a proxy for entity [com.atlassian.crowd.model.user.InternalUser] with id [491526] with two open sessions. -- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440 2023-07-09 19:50:18,617 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] clearSession clear session [ SessionImpl(1974468763<open>) ] 2023-07-09 19:50:18,617 ERROR [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual Could not process class com.atlassian.crowd.embedded.hibernate2.HibernateMembership: HibernateMembership{groupMember=null, id=589834, parentGroup=com.atlassian.crowd.model.group.InternalGroup@409ee6f2[id=425989,name=test-group,type=GROUP,active=true,description=<null>,lowerName=test-group,createdDate=2023-07-09 19:46:46.71,updatedDate=2023-07-09 19:46:46.71,directoryId=360450,externalId=<null>], userMember=com.atlassian.crowd.model.user.InternalUser@1fbf093e[id=491526,name=testuser1,createdDate=Sun Jul 09 19:46:46 IST 2023,updatedDate=Sun Jul 09 19:46:46 IST 2023,active=true,emailAddress=testuser1@test.com,firstName=,lastName=testuser1,displayName=testuser1,credential=com.atlassian.crowd.embedded.api.PasswordCredential@4305f260[credential=********,encryptedCredential=true],lowerName=testuser1,lowerEmailAddress=testuser1@test.com,lowerFirstName=,lowerLastName=testuser1,lowerDisplayName=testuser1,directoryId=360450,externalId=1:15f4409c-4abf-408a-9783-7ede09452550]} -- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440 javax.persistence.PersistenceException: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491526] with two open Sessions at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:188) at org.hibernate.internal.SessionImpl.fireDelete(SessionImpl.java:950) at org.hibernate.internal.SessionImpl.delete(SessionImpl.java:872) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:15) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:11) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.performOperation(AbstractBatchProcessor.java:217) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processIndividual(AbstractBatchProcessor.java:188) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processBatch(AbstractBatchProcessor.java:172) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:133) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:82) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeAll(HibernateMembershipDao.java:503) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeUsersFromGroup(HibernateMembershipDao.java:386) at com.atlassian.confluence.impl.user.crowd.CachedCrowdMembershipDao.removeUsersFromGroup(CachedCrowdMembershipDao.java:146) .... Caused by: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491526] with two open Sessions at org.hibernate.proxy.AbstractLazyInitializer.setSession(AbstractLazyInitializer.java:134)
And this cycle continues until all the records under the batch are processed. During this process, it is observed that during individual processing, Confluence only processes the first record from the batch, while the rest encounter the same "illegally attempted to associate proxy with two open Sessions" error.
However, despite the errors, Confluence marks this directory sync as completed.
This is reproducible on Data Center: (yes).
Note
Same issue also happens when: * Setup LDAP with nested group
- Remove multiple nested groups from parent group then synchronise
As LDAP is always using full sync which will process remove operation in batch
Steps to Reproduce
- Step 1: Create a Jira User Directory in Confluence.
- Step 2: In Jira, create two or more users and assign them to a group, let's call it "test_group" in Jira.
- Step 3: Enable the following debug packages in Confluence as TRACE
com.atlassian.confluence.impl.user.crowd.hibernate com.atlassian.confluence.impl.user.crowd.hibernate.batch
- Step 4: Sync this User Directory with Confluence. During this sync, Confluence processes all the records and creates corresponding users in Confluence.
- Step 5: Remove the "test_group" group from the two users that were created earlier in Jira.
- Step 6: Trigger the Jira User Directory sync again in Confluence, and you will encounter the same error mentioned earlier.
- Step 7: During each sync, Confluence removes one user from the group in Confluence as part of the individual processing.
Expected Results
The process of removing group membership for multiple users in a batch should be executed without encountering the error "Illegally attempted to associate a proxy with two open sessions." If the batch fails due to any other reason, the individual processing will still proceed and process all the records within the same batch without encountering any failure.
Actual Results
During user directory synchronization, the removal of user group membership encounters a failure in the batch operation "com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation". This failure is accompanied by the error message "Illegally attempted to associate a proxy with two open sessions."
2023-07-09 19:50:18,601 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] execute processing [ 2 ] objects in [ 1 ] batches of [ 100 ] with [ com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation ] 2023-07-09 19:50:18,601 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] openSession using session [ SessionImpl(1974468763<open>) ] 2023-07-09 19:50:18,601 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ] 2023-07-09 19:50:18,608 ERROR [Caesium-1-1] [org.hibernate.proxy.AbstractLazyInitializer] setSession HHH000485: Illegally attempted to associate a proxy for entity [com.atlassian.crowd.model.user.InternalUser] with id [491524] with two open sessions. -- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440 2023-07-09 19:50:18,613 WARN [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processBatch batch failed falling back to individual processing -- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440 javax.persistence.PersistenceException: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491524] with two open Sessions at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:188) at org.hibernate.internal.SessionImpl.fireDelete(SessionImpl.java:950) at org.hibernate.internal.SessionImpl.delete(SessionImpl.java:872) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:15) at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:11) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.performOperation(AbstractBatchProcessor.java:217) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processBatch(AbstractBatchProcessor.java:156) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:133) at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:82) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeAll(HibernateMembershipDao.java:503) at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeUsersFromGroup(HibernateMembershipDao.java:386) at com.atlassian.confluence.impl.user.crowd.CachedCrowdMembershipDao.removeUsersFromGroup(CachedCrowdMembershipDao.java:146)
Due to the aforementioned error, the system switches to individual processing. However, during the individual processing, only the first record is successfully processed, while the rest encounter the same error message.
2023-07-09 19:50:18,614 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual processing [ 2 ] individually
2023-07-09 19:50:18,614 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual processed [ HibernateMembership{groupMember=null, id=589833, parentGroup=com.atlassian.crowd.model.group.InternalGroup@409ee6f2[id=425989,name=test-group,type=GROUP,active=true,description=<null>,lowerName=test-group,createdDate=2023-07-09 19:46:46.71,updatedDate=2023-07-09 19:46:46.71,directoryId=360450,externalId=<null>], userMember=com.atlassian.crowd.model.user.InternalUser@3e045818[id=491524,name=testuser2,createdDate=Sun Jul 09 19:46:46 IST 2023,updatedDate=Sun Jul 09 19:46:46 IST 2023,active=true,emailAddress=testuser2@test.com,firstName=,lastName=testuser2,displayName=testuser2,credential=com.atlassian.crowd.embedded.api.PasswordCredential@783baecd[credential=********,encryptedCredential=true],lowerName=testuser2,lowerEmailAddress=testuser2@test.com,lowerFirstName=,lowerLastName=testuser2,lowerDisplayName=testuser2,directoryId=360450,externalId=1:f34de7f2-88d1-4fcd-a2e5-39323efd4819]} ] [ 50.0% ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] commitTransaction commit transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,615 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] flushSession flush session [ SessionImpl(1974468763<open>) ]
2023-07-09 19:50:18,616 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] clearSession clear session [ SessionImpl(1974468763<open>) ]
2023-07-09 19:50:18,617 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] beginTransaction begin transaction [ org.hibernate.engine.transaction.internal.TransactionImpl@359d64ac ]
2023-07-09 19:50:18,617 ERROR [Caesium-1-1] [org.hibernate.proxy.AbstractLazyInitializer] setSession HHH000485: Illegally attempted to associate a proxy for entity [com.atlassian.crowd.model.user.InternalUser] with id [491526] with two open sessions.
-- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440
2023-07-09 19:50:18,617 DEBUG [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] clearSession clear session [ SessionImpl(1974468763<open>) ]
2023-07-09 19:50:18,617 ERROR [Caesium-1-1] [crowd.hibernate.batch.Hibernate5BatchProcessor] processIndividual Could not process class com.atlassian.crowd.embedded.hibernate2.HibernateMembership: HibernateMembership{groupMember=null, id=589834, parentGroup=com.atlassian.crowd.model.group.InternalGroup@409ee6f2[id=425989,name=test-group,type=GROUP,active=true,description=<null>,lowerName=test-group,createdDate=2023-07-09 19:46:46.71,updatedDate=2023-07-09 19:46:46.71,directoryId=360450,externalId=<null>], userMember=com.atlassian.crowd.model.user.InternalUser@1fbf093e[id=491526,name=testuser1,createdDate=Sun Jul 09 19:46:46 IST 2023,updatedDate=Sun Jul 09 19:46:46 IST 2023,active=true,emailAddress=testuser1@test.com,firstName=,lastName=testuser1,displayName=testuser1,credential=com.atlassian.crowd.embedded.api.PasswordCredential@4305f260[credential=********,encryptedCredential=true],lowerName=testuser1,lowerEmailAddress=testuser1@test.com,lowerFirstName=,lowerLastName=testuser1,lowerDisplayName=testuser1,directoryId=360450,externalId=1:15f4409c-4abf-408a-9783-7ede09452550]}
-- url: /c823/setup/setupdata.action | userName: anonymous | action: setupdata | referer: http://localhost:6823/c823/setup/setupdata-start.action | traceId: dd89c22c7c19d440
javax.persistence.PersistenceException: org.hibernate.HibernateException: illegally attempted to associate proxy [com.atlassian.crowd.model.user.InternalUser#491526] with two open Sessions
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:188)
at org.hibernate.internal.SessionImpl.fireDelete(SessionImpl.java:950)
at org.hibernate.internal.SessionImpl.delete(SessionImpl.java:872)
at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:15)
at com.atlassian.confluence.impl.user.crowd.hibernate.batch.operation.RemoveOperation.performOperation(RemoveOperation.java:11)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.performOperation(AbstractBatchProcessor.java:217)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processIndividual(AbstractBatchProcessor.java:188)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.processBatch(AbstractBatchProcessor.java:172)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:133)
at com.atlassian.crowd.util.persistence.hibernate.batch.AbstractBatchProcessor.execute(AbstractBatchProcessor.java:82)
at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeAll(HibernateMembershipDao.java:503)
at com.atlassian.confluence.impl.user.crowd.hibernate.HibernateMembershipDao.removeUsersFromGroup(HibernateMembershipDao.java:386)
at com.atlassian.confluence.impl.user.crowd.CachedCrowdMembershipDao.removeUsersFromGroup(CachedCrowdMembershipDao.java:146)
Workaround
- This issue is not reproduced while performing the Increment user directory sync.
- Or we can remove the group membership for users who have changes in the source system such as Jira, Crowd, or AD, execute the following SQL query to fetch the relevant records from the cwd_membership table. Make sure to update the Directory_ID, Group Name, and user lower user name accordingly:
select hibernatem0_.id as id1_32_, hibernatem0_.parent_id as parent_i2_32_, hibernatem0_.child_group_id as child_gr3_32_, hibernatem0_.child_user_id as child_us4_32_ from cwd_membership hibernatem0_ cross join cwd_group internalgr1_ cross join cwd_user internalus3_ where hibernatem0_.parent_id = internalgr1_.id and hibernatem0_.child_user_id = internalus3_.id and internalgr1_.directory_id = <Directory_ID> and internalgr1_.lower_group_name = <Group_Name> and (internalus3_.lower_user_name in (? , ?));
Once we have those records, we can run the simple delete statement:
delete from cwd_membership where id=?
- mentioned in
-
-
Jeffery Xie
-
- Votes:
-
13 Vote for this issue
- Watchers:
-
36 Start watching this issue
- Created:
- Updated:
- Resolved: