Loading...

Type: Bug
Resolution: Unresolved
Priority: Low
Fix Version/s: None
Affects Version/s: 10.2.0, 10.3.0, 11.3.0
Component/s: Login
Labels:

Introduced in Version:
10.02
Support reference count:
1
Symptom Severity:
Severity 3 - Minor
UIS:
1

Issue Summary

Since Jira 10.2.0, requests authenticating using PAT operate under a lock, which causes all requests to execute not concurrently.

While it's introduction fixed legitimate problems, we consider it a stability/performance risk/problem since:

It introduces a bottleneck to nodes processing high volume of request using PAT, not present for other authentication types
If something bad happened to that single thread actively processing the PAT (EG latency/interuption to the database), then all other PAT requests must wait

Steps to Reproduce

Do a bunch of PAT authenticated requests
Observe that only a single thread can execute (process PAT) at a time

Expected Results

Multiple threads can process PAT

Actual Results

Only 1 thread can process PAT.

Example 1 runnable thread:

"http-nio-8080-exec-64 url: /rest/api/2/search" #2488 daemon prio=5 os_prio=0 cpu=2973498.83ms elapsed=175680.73s tid=0x00007f7e818d09c0 nid=0x3ec18 runnable  [0x00007f65fb518000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.Net.poll(java.base@17.0.15-beta/Native Method)
	at sun.nio.ch.NioSocketImpl.park(java.base@17.0.15-beta/NioSocketImpl.java:186)
	at sun.nio.ch.NioSocketImpl.park(java.base@17.0.15-beta/NioSocketImpl.java:195)
	at sun.nio.ch.NioSocketImpl.implRead(java.base@17.0.15-beta/NioSocketImpl.java:319)
	at sun.nio.ch.NioSocketImpl.read(java.base@17.0.15-beta/NioSocketImpl.java:355)
	at sun.nio.ch.NioSocketImpl$1.read(java.base@17.0.15-beta/NioSocketImpl.java:808)
	at java.net.Socket$SocketInputStream.read(java.base@17.0.15-beta/Socket.java:966)
	at sun.security.ssl.SSLSocketInputRecord.read(java.base@17.0.15-beta/SSLSocketInputRecord.java:484)
	at sun.security.ssl.SSLSocketInputRecord.readHeader(java.base@17.0.15-beta/SSLSocketInputRecord.java:478)
	at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(java.base@17.0.15-beta/SSLSocketInputRecord.java:70)
	at sun.security.ssl.SSLSocketImpl.readApplicationRecord(java.base@17.0.15-beta/SSLSocketImpl.java:1465)
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(java.base@17.0.15-beta/SSLSocketImpl.java:1069)
	at java.io.FilterInputStream.read(java.base@17.0.15-beta/FilterInputStream.java:132)
	at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:64)
	at com.mysql.cj.protocol.a.SimplePacketReader.readHeaderLocal(SimplePacketReader.java:81)
	at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
	at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
	at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:52)
	at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:41)
	at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:54)
	at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:44)
	at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:576)
	at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:762)
	at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:701)
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:1052)
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:998)
	at com.mysql.cj.NativeSession.execSQL(NativeSession.java:655)
	at com.mysql.cj.jdbc.ConnectionImpl.setTransactionIsolation(ConnectionImpl.java:2216)
	- locked <0x00007f74e6db06d0> (a com.mysql.cj.jdbc.ConnectionImpl)
	at org.apache.commons.dbcp2.DelegatingConnection.setTransactionIsolation(DelegatingConnection.java:989)
	at org.apache.commons.dbcp2.DelegatingConnection.setTransactionIsolation(DelegatingConnection.java:989)
	at org.ofbiz.core.entity.jdbc.interceptors.connection.DelegatingConnection.setTransactionIsolation(DelegatingConnection.java:93)
	at com.atlassian.jira.ofbiz.sql.ConnectionWrapper.setTransactionIsolation(ConnectionWrapper.java:112)
	at org.ofbiz.core.entity.TransactionUtil.beginLocalTransaction(TransactionUtil.java:289)
	at com.atlassian.core.ofbiz.util.CoreTransactionUtil.begin(CoreTransactionUtil.java:54)
	at com.atlassian.jira.transaction.TransactionSupportImpl.beginTxn(TransactionSupportImpl.java:52)
	at com.atlassian.jira.transaction.TransactionSupportImpl.startedTransaction(TransactionSupportImpl.java:46)
	at com.atlassian.jira.transaction.TransactionSupportImpl.begin(TransactionSupportImpl.java:31)
	at com.atlassian.jira.transaction.Txn.begin(Txn.java:17)
	at com.atlassian.jira.database.DatabaseAccessorImpl.runInManagedTransaction(DatabaseAccessorImpl.java:131)
	at jdk.internal.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.15-beta/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@17.0.15-beta/Method.java:569)
	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
	at jdk.proxy3.$Proxy281.runInManagedTransaction(jdk.proxy3/Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.15-beta/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@17.0.15-beta/Method.java:569)
	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:134)
	at jdk.proxy3.$Proxy281.runInManagedTransaction(jdk.proxy3/Unknown Source)
	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.runInStartedOrExistingTransaction(JiraHostConnectionAccessor.java:130)
	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.execute(JiraHostConnectionAccessor.java:60)
	at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.execute(DefaultTransactionalExecutor.java:65)
	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.execute(DatabaseAccessorImpl.java:67)
	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.runInTransaction(DatabaseAccessorImpl.java:43)
	at com.atlassian.data.activeobjects.repository.query.PocketKnifeQueryExecutions$SingleEntityExecution.doExecute(PocketKnifeQueryExecutions.java:111)
	at com.atlassian.data.activeobjects.repository.query.AbstractActiveObjectsQueryExecution.execute(AbstractActiveObjectsQueryExecution.java:39)
	at com.atlassian.data.activeobjects.repository.query.AbstractActiveObjectsQuery.doExecute(AbstractActiveObjectsQuery.java:62)
	at com.atlassian.data.activeobjects.repository.query.AbstractActiveObjectsQuery.execute(AbstractActiveObjectsQuery.java:55)
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryQueryMethodInvoker$$Lambda$7361/0x00007f6d639baa60.invoke(Unknown Source)
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:159)
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:138)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:80)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
	at jdk.proxy187.$Proxy4248.getByTokenIdAndExpiringAtIsAfter(jdk.proxy187/Unknown Source)
	at com.atlassian.pats.service.DefaultTokenAuthenticationService.retrieveToken(DefaultTokenAuthenticationService.java:61)
	at com.atlassian.pats.service.DefaultTokenAuthenticationService.authenticate(DefaultTokenAuthenticationService.java:48)
	at com.atlassian.pats.web.filter.TokenBasedAuthenticationFilter.authenticateUsingToken(TokenBasedAuthenticationFilter.java:111)
	at com.atlassian.pats.web.filter.TokenBasedAuthenticationFilter.doFilter(TokenBasedAuthenticationFilter.java:80)
    ...

Example N blocked threads:

"http-nio-8080-exec-63 url: /rest/api/2/serverInfo" #2487 daemon prio=5 os_prio=0 cpu=2953171.23ms elapsed=175680.73s tid=0x00007f7e818cfc70 nid=0x3ec17 waiting on condition  [0x00007f65fb61a000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.15-beta/Native Method)
	- parking to wait for  <0x00007f6e7984ba38> (a java.util.concurrent.locks.ReentrantLock$FairSync)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.15-beta/LockSupport.java:211)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.15-beta/AbstractQueuedSynchronizer.java:715)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.15-beta/AbstractQueuedSynchronizer.java:938)
	at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@17.0.15-beta/ReentrantLock.java:153)
	at java.util.concurrent.locks.ReentrantLock.lock(java.base@17.0.15-beta/ReentrantLock.java:322)
	at com.atlassian.pats.service.DefaultTokenAuthenticationService.retrieveToken(DefaultTokenAuthenticationService.java:59)
	at com.atlassian.pats.service.DefaultTokenAuthenticationService.authenticate(DefaultTokenAuthenticationService.java:48)
	at com.atlassian.pats.web.filter.TokenBasedAuthenticationFilter.authenticateUsingToken(TokenBasedAuthenticationFilter.java:111)
	at com.atlassian.pats.web.filter.TokenBasedAuthenticationFilter.doFilter(TokenBasedAuthenticationFilter.java:80)
    ...

Workaround

Use a different authentication technique (cookie, basic)

Have more nodes (lock isn't sync across cluster)

is a regression of

JRASERVER-76340 Scripts failing intermittently due to permissions denied (401) exception while using PAT

Closed

Details

Description

Issue Summary

Steps to Reproduce

Expected Results

Actual Results

Workaround

Attachments

Issue Links

Activity

People

Dates