[JRASERVER-71876] DB connection failure causing threads stuck on user authentication

Type: Bug
Resolution: Fixed
Priority: High (View bug fix roadmap)
Fix Version/s: 8.16.0
Affects Version/s: 8.13.0
Component/s: Environment - Database, Login, User Management - Others
Labels:

Introduced in Version:
8.13
Support reference count:
4
Symptom Severity:
Severity 2 - Major
UIS:
11
Bug Fix Policy:
View Atlassian Server bug fix policy
Current Status:

Hide

Hi All,

We have some updates related to current bug. It seems to be related to a broader issue with database connectivity: ~~JRASERVER-62072~~.

It turned out that solution introduced in this ticket can cause other problems with long queries to the database, for example during upgrade tasks (please see: ~~JRASERVER-74314~~).
Unfortunately, we will revert changes from this ticket in upcoming Jira 9.4 to make sure upgrade tasks are working reliably.

For further information about database connectivity problems, please see mentioned issue: ~~JRASERVER-62072~~

Regards,
Michał, Jira Dev team

Show
Hi All, We have some updates related to current bug. It seems to be related to a broader issue with database connectivity: JRASERVER-62072 . It turned out that solution introduced in this ticket can cause other problems with long queries to the database, for example during upgrade tasks (please see: JRASERVER-74314 ). Unfortunately, we will revert changes from this ticket in upcoming Jira 9.4 to make sure upgrade tasks are working reliably. For further information about database connectivity problems, please see mentioned issue: JRASERVER-62072 Regards, Michał, Jira Dev team

Issue Summary

DB connection failure can cause a thread to be stuck during user authentication while thread is updating data into user's attributes.
This can lead to all other threads that need to authenticate to be blocked by the stuck thread, causing the entire application/node to crumble

Steps to Reproduce

Start user authentication thread
Break DB connection
start other user authentication threads

Expected Results

Initial thread is killed with an error message about the DB failure.
No other thread is affected

Actual Results

Initial thread gets stuck and locks the update to user's attributes blocking any other thread from successfully authenticating users.

Initial thread stuck:

"http-nio-8080-exec-111 url:/jira/rest/emai.../mail-queue/status" #20686 daemon prio=5 os_prio=0 tid=0x00007f98c0355000 nid=0x4aa4 runnable [0x00007f98796d0000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
	- locked <0x00000006ccae2808> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	- locked <0x00000006ccae4910> (a sun.security.ssl.AppInputStream)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
	at org.postgresql.core.PGStream.receiveChar(PGStream.java:321)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1978)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
	- locked <0x00000006ccae4948> (a org.postgresql.core.v3.QueryExecutorImpl)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:149)
	at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:124)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
	at com.atlassian.jira.ofbiz.sql.PreparedStatementWrapper.executeUpdate(PreparedStatementWrapper.java:47)
	at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement.lambda$executeUpdate$7(DiagnosticPreparedStatement.java:69)
	at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement$$Lambda$1523/887178670.execute(Unknown Source)
	at com.atlassian.diagnostics.internal.platform.monitor.db.DefaultDatabaseDiagnosticsCollector.recordExecutionTime(DefaultDatabaseDiagnosticsCollector.java:69)
	at com.atlassian.jira.diagnostic.connection.DatabaseDiagnosticsCollectorDelegate.recordExecutionTime(DatabaseDiagnosticsCollectorDelegate.java:55)
	at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement.executeUpdate(DiagnosticPreparedStatement.java:69)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:562)
	at org.ofbiz.core.entity.GenericDAO.singleUpdate(GenericDAO.java:358)
	at org.ofbiz.core.entity.GenericDAO.customUpdate(GenericDAO.java:286)
	at org.ofbiz.core.entity.GenericDAO.update(GenericDAO.java:266)
	at org.ofbiz.core.entity.GenericDAO.update(GenericDAO.java:236)
	at org.ofbiz.core.entity.GenericHelperDAO.store(GenericHelperDAO.java:257)
	at org.ofbiz.core.entity.GenericDelegator.store(GenericDelegator.java:1744)
	at org.ofbiz.core.entity.GenericDelegator.store(GenericDelegator.java:1725)
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.store(DefaultOfBizDelegator.java:281)
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.store(WrappingOfBizDelegator.java:166)
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.storeUser(OfBizUserDao.java:512)
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.update(OfBizUserDao.java:496)
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.update(OfBizUserDao.java:480)
	at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.update(DelegatingUserDao.java:97)
	at com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.update(IndexedUserDao.java:326)
	- locked <0x00000006c3d70fa8> (a java.lang.Object)
	at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.update(DelegatingUserDao.java:97)
	at com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.update(SwitchingUserDao.java:30)
	at com.atlassian.crowd.directory.CachingDirectory.updateUser(CachingDirectory.java:144)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateUserAndSetActiveFlag(DbCachingRemoteDirectory.java:340)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateUserFromRemoteDirectory(DbCachingRemoteDirectory.java:279)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticateAndUpdateInternalUser(DbCachingRemoteDirectory.java:257)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticate(DbCachingRemoteDirectory.java:174)
	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:304)
	at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:177)
	at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:70)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.crowdServiceAuthenticate(JiraSeraphAuthenticator.java:75)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.authenticate(JiraSeraphAuthenticator.java:49)
	at com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:90)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.getUserFromBasicAuthentication(JiraSeraphAuthenticator.java:131)
	at com.atlassian.seraph.auth.DefaultAuthenticator.getUser(DefaultAuthenticator.java:341)
	...

All other threads blocked by initial thread:

"http-nio-8080-exec-539 url:/jira/rest/emai.../mail-queue/status" #88194 daemon prio=5 os_prio=0 tid=0x00007f98c02ae000 nid=0x7d78 waiting for monitor entry [0x00007f9871ac2000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.update(IndexedUserDao.java:324)
	- waiting to lock <0x00000006c3d70fa8> (a java.lang.Object)
	at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.update(DelegatingUserDao.java:97)
	at com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.update(SwitchingUserDao.java:30)
	at com.atlassian.crowd.directory.CachingDirectory.updateUser(CachingDirectory.java:144)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateUserAndSetActiveFlag(DbCachingRemoteDirectory.java:340)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.updateUserFromRemoteDirectory(DbCachingRemoteDirectory.java:279)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticateAndUpdateInternalUser(DbCachingRemoteDirectory.java:257)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticate(DbCachingRemoteDirectory.java:174)
	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:304)
	at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:177)
	at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:70)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.crowdServiceAuthenticate(JiraSeraphAuthenticator.java:75)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.authenticate(JiraSeraphAuthenticator.java:49)
	at com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:90)
	at com.atlassian.jira.security.login.JiraSeraphAuthenticator.getUserFromBasicAuthentication(JiraSeraphAuthenticator.java:131)
	at com.atlassian.seraph.auth.DefaultAuthenticator.getUser(DefaultAuthenticator.java:341)
	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:139)

Workaround

Restart node

is related to

JRASERVER-62072 Database connectivity issue causes scheduled jobs to break

Closed

HL-1903 You do not have permission to view this issue

relates to

JRASERVER-71982 Upgrade PostgresSQL JDBC driver to 42.2.16+ version

Closed

causes: PS-69325 You do not have permission to view this issue

mentioned in: Page Failed to load; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

(6 mentioned in)

Form Name

Michał Błajet added a comment - 04/Nov/2022 11:32 AM

This bug seems to be related to a broader issue with database connectivity: https://jira.atlassian.com/browse/JRASERVER-62072. It turned out that solution introduced in this ticket can cause other problems with long queries to the database, for example during upgrade tasks (please see: https://jira.atlassian.com/browse/JRASERVER-74314). We will revert changes from this ticket in upcoming Jira 9.4. For further information about database connectivity problems, please see mentioned issue: https://jira.atlassian.com/browse/JRASERVER-62072

Michał Błajet added a comment - 04/Nov/2022 11:32 AM This bug seems to be related to a broader issue with database connectivity: https://jira.atlassian.com/browse/JRASERVER-62072 . It turned out that solution introduced in this ticket can cause other problems with long queries to the database, for example during upgrade tasks (please see: https://jira.atlassian.com/browse/JRASERVER-74314 ). We will revert changes from this ticket in upcoming Jira 9.4. For further information about database connectivity problems, please see mentioned issue: https://jira.atlassian.com/browse/JRASERVER-62072

Andriy Yakovlev [Atlassian] added a comment - 28/Jun/2021 8:52 AM

Hey 3d67332d16d7
This problem is a rare edge case related to network instability.
You can you want to cover this in 8.13, you can updgrade driver manually and adjust Timeout settings, see ~~JRASERVER-71982~~ for more details.

Andriy Yakovlev [Atlassian] added a comment - 28/Jun/2021 8:52 AM Hey 3d67332d16d7 This problem is a rare edge case related to network instability. You can you want to cover this in 8.13, you can updgrade driver manually and adjust Timeout settings, see JRASERVER-71982 for more details.

Gonchik Tsymzhitov added a comment - 25/Jun/2021 8:56 PM

Do you know situation with 8.13.x ?

Does it enough just upgrade driver to latest version?

Gonchik Tsymzhitov added a comment - 25/Jun/2021 8:56 PM Do you know situation with 8.13.x ? Does it enough just upgrade driver to latest version?

Details

Description

Issue Summary

Steps to Reproduce

Expected Results

Actual Results

Workaround

Attachments

Issue Links

Forms

Activity

Collapse comment: Michał Błajet added a comment - 04/Nov/2022 11:32 AM

Expand comment: Michał Błajet added a comment - 04/Nov/2022 11:32 AM

Collapse comment: Andriy Yakovlev [Atlassian] added a comment - 28/Jun/2021 8:52 AM

Expand comment: Andriy Yakovlev [Atlassian] added a comment - 28/Jun/2021 8:52 AM

Collapse comment: Gonchik Tsymzhitov added a comment - 25/Jun/2021 8:56 PM

Expand comment: Gonchik Tsymzhitov added a comment - 25/Jun/2021 8:56 PM

People

Dates