Loading...

Details

Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 8.16.0
Affects Version/s: 8.13.0
Component/s: Environment - Database, Login, User Management - Others
Labels:

Introduced in Version:
8.13
Support reference count:
4
Symptom Severity:
Severity 2 - Major
UIS:
11
Bug Fix Policy:
View Atlassian Server bug fix policy
Current Status:

Hide

Hi All,

We have some updates related to current bug. It seems to be related to a broader issue with database connectivity: ~~JRASERVER-62072~~.

It turned out that solution introduced in this ticket can cause other problems with long queries to the database, for example during upgrade tasks (please see: ~~JRASERVER-74314~~).
Unfortunately, we will revert changes from this ticket in upcoming Jira 9.4 to make sure upgrade tasks are working reliably.

For further information about database connectivity problems, please see mentioned issue: ~~JRASERVER-62072~~

Regards,
Michał, Jira Dev team

Show
Hi All, We have some updates related to current bug. It seems to be related to a broader issue with database connectivity: JRASERVER-62072 . It turned out that solution introduced in this ticket can cause other problems with long queries to the database, for example during upgrade tasks (please see: JRASERVER-74314 ). Unfortunately, we will revert changes from this ticket in upcoming Jira 9.4 to make sure upgrade tasks are working reliably. For further information about database connectivity problems, please see mentioned issue: JRASERVER-62072 Regards, Michał, Jira Dev team

Description

Issue Summary

DB connection failure can cause a thread to be stuck during user authentication while thread is updating data into user's attributes.
This can lead to all other threads that need to authenticate to be blocked by the stuck thread, causing the entire application/node to crumble

Steps to Reproduce

Start user authentication thread
Break DB connection
start other user authentication threads

Expected Results

Initial thread is killed with an error message about the DB failure.
No other thread is affected

Actual Results

Initial thread gets stuck and locks the update to user's attributes blocking any other thread from successfully authenticating users.

Initial thread stuck:

"http-nio-8080-exec-111 url:/jira/rest/emai.../mail-queue/status" #20686 daemon prio=5 os_prio=0 tid=0x00007f98c0355000 nid=0x4aa4 runnable [0x00007f98796d0000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
	- locked <0x00000006ccae2808> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	- locked <0x00000006ccae4910> (a sun.security.ssl.AppInputStream)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
	at org.postgresql.core.PGStream.receiveChar(PGStream.java:321)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1978)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
	- locked <0x00000006ccae4948> (a org.postgresql.core.v3.QueryExecutorImpl)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:149)
	at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:124)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
	at com.atlassian.jira.ofbiz.sql.PreparedStatementWrapper.executeUpdate(PreparedStatementWrapper.java:47)
	at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement.lambda$executeUpdate$7(DiagnosticPreparedStatement.java:69)
	at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement$$Lambda$1523/887178670.execute(Unknown Source)
	at com.atlassian.diagnostics.internal.platform.monitor.db.DefaultDatabaseDiagnosticsCollector.recordExecutionTime(DefaultDatabaseDiagnosticsCollector.java:69)
	at com.atlassian.jira.diagnostic.connection.DatabaseDiagnosticsCollectorDelegate.recordExecutionTime(DatabaseDiagnosticsCollectorDelegate.java:55)
	at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement.executeUpdate(DiagnosticPreparedStatement.java:69)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:562)
	at org.ofbiz.core.entity.GenericDAO.singleUpdate(GenericDAO.java:358)
	at org.ofbiz.core.entity.GenericDAO.customUpdate(GenericDAO.java:286)
	at org.ofbiz.core.entity.GenericDAO.update(GenericDAO.java:266)
	at org.ofbiz.core.entity.GenericDAO.update(GenericDAO.java:236)
	at org.ofbiz.core.entity.GenericHelperDAO.store(GenericHelperDAO.java:257)
	at org.ofbiz.core.entity.GenericDelegator.store(GenericDelegator.java:1744)
	at org.ofbiz.core.entity.GenericDelegator.store(GenericDelegator.java:1725)
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.store(DefaultOfBizDelegator.java:281)
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.store(WrappingOfBizDelegator.java:166)
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.storeUser(OfBizUserDao.java:512)
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.update(OfBizUserDao.java:496)
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.update(OfBizUserDao.java:480)
	at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.update(DelegatingUserDao.java:97)
	at com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.update(IndexedUserDao.java:326)
	- locked <0x00000006c3d70fa8> (a java.lang.Object)
	at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.update(DelegatingUserDao.java:97)
	at com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.update(SwitchingUserDao.java:30)
	at com.atlassian.crowd.directory.CachingDirectory.updateUser(CachingDirectory.java:144)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateUserAndSetActiveFlag(DbCachingRemoteDirectory.java:340)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateUserFromRemoteDirectory(DbCachingRemoteDirectory.java:279)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticateAndUpdateInternalUser(DbCachingRemoteDirectory.java:257)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticate(DbCachingRemoteDirectory.java:174)
	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:304)
	at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:177)
	at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:70)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.crowdServiceAuthenticate(JiraSeraphAuthenticator.java:75)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.authenticate(JiraSeraphAuthenticator.java:49)
	at com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:90)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.getUserFromBasicAuthentication(JiraSeraphAuthenticator.java:131)
	at com.atlassian.seraph.auth.DefaultAuthenticator.getUser(DefaultAuthenticator.java:341)
	...

All other threads blocked by initial thread:

"http-nio-8080-exec-539 url:/jira/rest/emai.../mail-queue/status" #88194 daemon prio=5 os_prio=0 tid=0x00007f98c02ae000 nid=0x7d78 waiting for monitor entry [0x00007f9871ac2000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.update(IndexedUserDao.java:324)
	- waiting to lock <0x00000006c3d70fa8> (a java.lang.Object)
	at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.update(DelegatingUserDao.java:97)
	at com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.update(SwitchingUserDao.java:30)
	at com.atlassian.crowd.directory.CachingDirectory.updateUser(CachingDirectory.java:144)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateUserAndSetActiveFlag(DbCachingRemoteDirectory.java:340)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateUserFromRemoteDirectory(DbCachingRemoteDirectory.java:279)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticateAndUpdateInternalUser(DbCachingRemoteDirectory.java:257)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticate(DbCachingRemoteDirectory.java:174)
	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:304)
	at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:177)
	at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:70)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.crowdServiceAuthenticate(JiraSeraphAuthenticator.java:75)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.authenticate(JiraSeraphAuthenticator.java:49)
	at com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:90)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.getUserFromBasicAuthentication(JiraSeraphAuthenticator.java:131)
	at com.atlassian.seraph.auth.DefaultAuthenticator.getUser(DefaultAuthenticator.java:341)
	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:139)

Workaround

Restart node

Attachments

Issue Links

is related to

JRASERVER-62072 Database connectivity issue causes scheduled jobs to break

Closed

HL-1903 Loading...

relates to

JRASERVER-71982 Upgrade PostgresSQL JDBC driver to 42.2.16+ version

Closed

causes: PS-69325 Loading...

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

(6 mentioned in)

DB connection failure causing threads stuck on user authentication