Loading...

Details

Type: Bug
Resolution: Not a bug
Priority: High
Fix Version/s: None
Affects Version/s: 4.9.1, 4.13.0, 4.14.4, 5.7.1
Component/s: API - REST
Labels:
- dob-pr

Support reference count:
31
Symptom Severity:
Severity 3 - Minor
UIS:
22
Bug Fix Policy:
View Atlassian Server bug fix policy

Description

Summary

If REST API calls are made in parallel (or in very rapid succession) a 401 is returned and a SQL exception is seen in the logs.

Environment

Bitbucket Server 4.9.1, 4.13.0
Crowd 2.8.3

Steps to Reproduce

Create a new group in Crowd
Add a Bitbucket Server admin user to the new group in Crowd
Ensure "Update group memberships when logging in - Every time the user logs in" is enabled
Use a script to hit 2 endpoints in parallel or rapid succession and authenticate as the user you added to the new group in Crowd

'http://<base_url>/rest/api/1.0/admin/permissions/groups'
'http://<base_url>/rest/api/1.0/admin/users/more-members?limit=200&context=svc-zptools&start=0'

Expected Results

Both calls should elicit 200 responses and no DB error should be seen in the logs

Actual Results

The below exception is thrown in the atlassian-bitbucket-access.log:

192.XXX.X.X,0:0:0:0:0:0:0:1 | http | o@MZCTPFx1330x25536x1 | - | 2017-03-07 22:10:39,797 | "GET /rest/api/1.0/admin/permissions/groups HTTP/1.1" | "" "" | 401 | 0 | 152 | - | 214 | - | 
192.XXX.X.X,0:0:0:0:0:0:0:1 | http | o@MZCTPFx1330x25535x0 | bcrusher | 2017-03-07 22:10:39,808 | "GET /rest/api/1.0/admin/users/more-members HTTP/1.1" | "" "" | 200 | 0 | 62 | - | 226 | w0ant7 |

The following is seen in the atlassian-bitbucket.log:

2017-03-07 22:10:39,740 DEBUG [http-nio-7990-exec-5] @MZCTPFx1330x25535x0 192.168.7.1,0:0:0:0:0:0:0:1 "GET /rest/api/1.0/admin/users/more-members HTTP/1.1" c.a.s.i.crowd.HibernateMembershipDao Saving object: com.atlassian.crowd.model.membership.InternalMembership@31bda479[parentId=3670018,childId=3277996,membershipType=GROUP_USER,groupType=GROUP,parentName=enterprise,lowerParentName=enterprise,childName=bcrusher,lowerChildName=bcrusher,directoryId=3211266]
2017-03-07 22:10:39,742 ERROR [http-nio-7990-exec-9] @MZCTPFx1330x25536x1 192.168.7.1,0:0:0:0:0:0:0:1 "GET /rest/api/1.0/admin/permissions/groups HTTP/1.1" o.h.e.j.batch.internal.BatchingBatch HHH000315: Exception executing batch [could not execute batch]
2017-03-07 22:10:39,768 WARN  [http-nio-7990-exec-9] @MZCTPFx1330x25536x1 192.168.7.1,0:0:0:0:0:0:0:1 "GET /rest/api/1.0/admin/permissions/groups HTTP/1.1" c.a.s.i.a.DefaultAuthenticationService Authenticator 'com.atlassian.bitbucket.server.bitbucket-authentication:crowdHttpAuthHandler' threw an exception
com.atlassian.bitbucket.DataStoreException: A database error has occurred.
	at com.atlassian.stash.internal.aop.ExceptionRewriteAdvice.afterThrowing(ExceptionRewriteAdvice.java:46) ~[bitbucket-platform-4.13.0.jar:na]
	at com.atlassian.stash.internal.auth.DefaultAuthenticationService.authenticateInternal(DefaultAuthenticationService.java:116) [bitbucket-service-impl-4.13.0.jar:na]
	at com.atlassian.stash.internal.auth.DefaultAuthenticationService.authenticate(DefaultAuthenticationService.java:67) [bitbucket-service-impl-4.13.0.jar:na]
	at com.atlassian.stash.internal.spring.security.PluginAuthenticationProvider.authenticate(PluginAuthenticationProvider.java:49) [classes/:na]
	at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:102) [classes/:na]
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112) [classes/:na]
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75) [classes/:na]
	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94) [atlassian-trusted-apps-core-4.2.0.jar:na]
	at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67) [atlassian-oauth-service-provider-plugin-2.0.3_1484801340000.jar:na]
	at com.atlassian.core.filters.ServletContextThreadLocalFilter.doFilter(ServletContextThreadLocalFilter.java:21) [atlassian-core-4.6.19.jar:na]
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) [atlassian-core-4.6.19.jar:na]
	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42) [classes/:na]
	at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109) [atlassian-connect-plugin-1.1.86-bitbucket-04.jar:na]
	at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:32) [jwt-plugin-1.5.11-0002_1484801340000.jar:na]
	at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:38) [analytics-client-5.2.20_1484801340000.jar:na]
	at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39) [analytics-client-5.2.20_1484801340000.jar:na]
--
	at com.atlassian.crowd.directory.AbstractInternalDirectory.addGroup(AbstractInternalDirectory.java:814) ~[crowd-persistence-2.10.2-rc04.jar:na]
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.syncRemoteGroupToInternalDirectory(DbCachingRemoteDirectory.java:470) ~[crowd-core-2.10.2-rc04.jar:na]
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateGroupsMembershipOnLogin(DbCachingRemoteDirectory.java:429) ~[crowd-core-2.10.2-rc04.jar:na]
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateUserFromRemoteDirectory(DbCachingRemoteDirectory.java:307) ~[crowd-core-2.10.2-rc04.jar:na]
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticateAndUpdateInternalUser(DbCachingRemoteDirectory.java:273) ~[crowd-core-2.10.2-rc04.jar:na]
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticate(DbCachingRemoteDirectory.java:183) ~[crowd-core-2.10.2-rc04.jar:na]
	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:311) ~[crowd-core-2.10.2-rc04.jar:na]
	at com.atlassian.stash.internal.crowd.CustomizedDirectoryManager.authenticateUser(CustomizedDirectoryManager.java:53) ~[bitbucket-service-impl-4.13.0.jar:na]
	at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:198) ~[crowd-core-2.10.2-rc04.jar:na]
	at com.atlassian.stash.internal.crowd.CustomizedApplicationService.authenticateUser(CustomizedApplicationService.java:50) ~[bitbucket-service-impl-4.13.0.jar:na]
	at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:75) ~[embedded-crowd-core-2.10.2-rc04.jar:na]
	at com.atlassian.stash.internal.crowd.RiotPolice.authenticate(RiotPolice.java:109) ~[bitbucket-service-impl-4.13.0.jar:na]
	at com.atlassian.stash.internal.plugin.OsgiSafeProxyProvider$1.invoke(OsgiSafeProxyProvider.java:93) ~[bitbucket-platform-4.13.0.jar:na]
	at com.atlassian.stash.internal.auth.EmbeddedCrowdHttpAuthenticationHandler.authenticate(EmbeddedCrowdHttpAuthenticationHandler.java:51) ~[bitbucket-service-impl-4.13.0.jar:na]
	at com.atlassian.stash.internal.auth.DefaultAuthenticationService.lambda$authenticateInternal$2(DefaultAuthenticationService.java:108) [bitbucket-service-impl-4.13.0.jar:na]
	at com.atlassian.stash.internal.auth.DefaultCaptchaService.authenticateWithCaptcha(DefaultCaptchaService.java:73) ~[bitbucket-service-impl-4.13.0.jar:na]
	... 23 common frames omitted
Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch
	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:129) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:136) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:114) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:163) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:226) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:484) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:351) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:350) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1258) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:425) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.beforeTransactionCommit(JdbcTransaction.java:101) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:177) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Final]
	at org.springframework.orm.hibernate4.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:584) ~[spring-orm-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	... 41 common frames omitted
Caused by: java.sql.BatchUpdateException: Duplicate entry 'enterprise-3211266' for key 'SYS_IDX_SYS_CT_10149_10151'
	at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1809) ~[mysql-connector-java-5.1.36-bin.jar:5.1.36]
	at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1271) ~[mysql-connector-java-5.1.36-bin.jar:5.1.36]
	at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128) ~[HikariCP-2.4.7.jar:na]
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) ~[HikariCP-2.4.7.jar:na]
	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:127) ~[hibernate-core-4.3.11.Final.jar:4.3.11.Fina

Notes

Seems there is a race condition where the fist API call authenticates and updates the group for the user while second call tries to do the same thing immediately after and a constraint violation occurs along with the user associated with the second call getting a 401.

Workaround

A. Disable this setting in the External User Directory settings in the Bitbucket Server UI
B. Run the script again (since the group has already been updated this should only occur on the initial parallel execution)
C. Run the calls in sequence
D. Create a user in the Bitbucket Server Internal User Directory and use that user for authentication with the REST API

Attachments

Issue Links

incorporates

BSERV-8171 Cache LDAP credentials in Bitbucket Server

Closed

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

(2 mentioned in)

401 response and SQL exception thrown when REST API calls are made in parallel