• Icon: Bug Bug
    • Resolution: Not a bug
    • Icon: High High
    • None
    • 2.7.2
    • Database
    • None
    • Ubuntu 14.04, OpenJDK 7, MySQL

      I see lots of exceptions like the one below.
      System seems very unstable. And it seems it's affecting the other systems (JIRA, Confluence, Bamboo, Stash).
      I turned token database off, and configured in memory.

      WARN  05:05:41,147 nTaskTimer ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@554eadfe -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
      WARN  05:05:41,150 nTaskTimer ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@554eadfe -- APPARENT DEADLOCK!!! Complete Status:
              Managed Threads: 3
              Active Threads: 3
              Active Tasks:
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@34dddfbe
                              on thread: C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#1
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@528c4e0f
                              on thread: C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#2
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7958e51f
                              on thread: C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#0
              Pending Tasks:
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3b2a9476
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@6878ed52
                      com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5b24d813
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@4fe5f624
                      com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@282de77
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5ddb57ea
                      com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6b68f3c9
                      com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@375d4836
                      com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4d2fed9f
                      com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5998950
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@2305e60f
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5842f4bf
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@13d254de
                      com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@55c1c4fc
                      com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@246ba860
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@17fb494e
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7289ac2e
      Pool thread stack traces:
              Thread[C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#0,5,main]
                      java.net.SocketInputStream.socketRead0(Native Method)
                      java.net.SocketInputStream.read(SocketInputStream.java:152)
                      java.net.SocketInputStream.read(SocketInputStream.java:122)
                      com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
                      com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
                      com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
                      com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
                      com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:598)
                      com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1086)
                      com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
                      com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2344)
                      com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
                      com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                      com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                      sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
                      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                      java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                      com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
                      com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                      com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                      com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                      com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                      com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                      com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                      com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                      com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
              Thread[C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#1,5,main]
                      java.net.SocketInputStream.socketRead0(Native Method)
                      java.net.SocketInputStream.read(SocketInputStream.java:152)
                      java.net.SocketInputStream.read(SocketInputStream.java:122)
                      com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
                      com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
                      com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
                      com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
                      com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:598)
                      com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1086)
                      com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
                      com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2344)
                      com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
                      com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                      com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                      sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
                      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                      java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                      com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
                      com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                      com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                      com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                      com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                      com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                      com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                      com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                      com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
              Thread[C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#2,5,main]
                      java.net.SocketInputStream.socketRead0(Native Method)
                      java.net.SocketInputStream.read(SocketInputStream.java:152)
                      java.net.SocketInputStream.read(SocketInputStream.java:122)
                      com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
                      com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
                      com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
                      com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
                      com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:598)
                      com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1086)
                      com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
                      com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2344)
                      com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
                      com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                      com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                      sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
                      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                      java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                      com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
                      com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                      com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                      com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                      com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                      com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                      com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                      com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                      com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                      com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
      

            [CWD-3915] APPARENT DEADLOCK exception

            Monique Khairuliana (Inactive) made changes -
            Workflow Original: Simplified Crowd Development Workflow v2 - restricted [ 1509743 ] New: JAC Bug Workflow v3 [ 3364036 ]
            Status Original: Resolved [ 5 ] New: Closed [ 6 ]
            Owen made changes -
            Workflow Original: Simplified Crowd Development Workflow v2 [ 1392045 ] New: Simplified Crowd Development Workflow v2 - restricted [ 1509743 ]
            Owen made changes -
            Workflow Original: Crowd Development Workflow v2 [ 686667 ] New: Simplified Crowd Development Workflow v2 [ 1392045 ]
            Jonathan G made changes -
            Remote Link Original: This issue links to "Page (Extranet)" [ 65533 ] New: This issue links to "Page (Extranet)" [ 65533 ]
            Caspar Krieger (Inactive) made changes -
            Remote Link Original: This issue links to "Page (Extranet)" [ 65533 ] New: This issue links to "Page (Extranet)" [ 65533 ]
            Caspar Krieger (Inactive) made changes -
            Remote Link Original: This issue links to "Page (Extranet)" [ 65533 ] New: This issue links to "Page (Extranet)" [ 65533 ]
            Diego Berrueta made changes -
            Resolution New: Not a bug [ 12 ]
            Status Original: Open [ 1 ] New: Resolved [ 5 ]
            Diego Berrueta made changes -
            Remote Link New: This issue links to "Page (Extranet)" [ 65533 ]
            Diego Berrueta made changes -
            Description Original: I see lots of exceptions like the one below.
            System seems very unstable. And it seems it's affecting the other systems (JIRA, Confluence, Bamboo, Stash).
            I turned token database off, and configured in memory.



            WARN 05:05:41,147 nTaskTimer ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@554eadfe -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
            WARN 05:05:41,150 nTaskTimer ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@554eadfe -- APPARENT DEADLOCK!!! Complete Status:
                    Managed Threads: 3
                    Active Threads: 3
                    Active Tasks:
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@34dddfbe
                                    on thread: C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#1
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@528c4e0f
                                    on thread: C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#2
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7958e51f
                                    on thread: C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#0
                    Pending Tasks:
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3b2a9476
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@6878ed52
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5b24d813
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@4fe5f624
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@282de77
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5ddb57ea
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6b68f3c9
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@375d4836
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4d2fed9f
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5998950
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@2305e60f
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5842f4bf
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@13d254de
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@55c1c4fc
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@246ba860
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@17fb494e
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7289ac2e
            Pool thread stack traces:
                    Thread[C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#0,5,main]
                            java.net.SocketInputStream.socketRead0(Native Method)
                            java.net.SocketInputStream.read(SocketInputStream.java:152)
                            java.net.SocketInputStream.read(SocketInputStream.java:122)
                            com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
                            com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
                            com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
                            com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
                            com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:598)
                            com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1086)
                            com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
                            com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2344)
                            com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
                            com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                            com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                            sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
                            sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                            java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                            com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
                            com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                            com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                            com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                            com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                            com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
                    Thread[C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#1,5,main]
                            java.net.SocketInputStream.socketRead0(Native Method)
                            java.net.SocketInputStream.read(SocketInputStream.java:152)
                            java.net.SocketInputStream.read(SocketInputStream.java:122)
                            com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
                            com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
                            com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
                            com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
                            com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:598)
                            com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1086)
                            com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
                            com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2344)
                            com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
                            com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                            com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                            sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
                            sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                            java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                            com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
                            com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                            com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                            com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                            com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                            com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
                    Thread[C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#2,5,main]
                            java.net.SocketInputStream.socketRead0(Native Method)
                            java.net.SocketInputStream.read(SocketInputStream.java:152)
                            java.net.SocketInputStream.read(SocketInputStream.java:122)
                            com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
                            com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
                            com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
                            com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
                            com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:598)
                            com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1086)
                            com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
                            com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2344)
                            com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
                            com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                            com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                            sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
                            sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                            java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                            com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
                            com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                            com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                            com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                            com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                            com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)

            New: I see lots of exceptions like the one below.
            System seems very unstable. And it seems it's affecting the other systems (JIRA, Confluence, Bamboo, Stash).
            I turned token database off, and configured in memory.

            {noformat}
            WARN 05:05:41,147 nTaskTimer ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@554eadfe -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
            WARN 05:05:41,150 nTaskTimer ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@554eadfe -- APPARENT DEADLOCK!!! Complete Status:
                    Managed Threads: 3
                    Active Threads: 3
                    Active Tasks:
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@34dddfbe
                                    on thread: C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#1
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@528c4e0f
                                    on thread: C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#2
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7958e51f
                                    on thread: C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#0
                    Pending Tasks:
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3b2a9476
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@6878ed52
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5b24d813
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@4fe5f624
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@282de77
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5ddb57ea
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6b68f3c9
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@375d4836
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4d2fed9f
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5998950
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@2305e60f
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5842f4bf
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@13d254de
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@55c1c4fc
                            com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@246ba860
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@17fb494e
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7289ac2e
            Pool thread stack traces:
                    Thread[C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#0,5,main]
                            java.net.SocketInputStream.socketRead0(Native Method)
                            java.net.SocketInputStream.read(SocketInputStream.java:152)
                            java.net.SocketInputStream.read(SocketInputStream.java:122)
                            com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
                            com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
                            com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
                            com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
                            com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:598)
                            com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1086)
                            com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
                            com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2344)
                            com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
                            com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                            com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                            sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
                            sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                            java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                            com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
                            com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                            com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                            com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                            com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                            com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
                    Thread[C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#1,5,main]
                            java.net.SocketInputStream.socketRead0(Native Method)
                            java.net.SocketInputStream.read(SocketInputStream.java:152)
                            java.net.SocketInputStream.read(SocketInputStream.java:122)
                            com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
                            com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
                            com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
                            com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
                            com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:598)
                            com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1086)
                            com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
                            com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2344)
                            com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
                            com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                            com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                            sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
                            sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                            java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                            com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
                            com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                            com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                            com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                            com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                            com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
                    Thread[C3P0PooledConnectionPoolManager[identityToken->itn5ok911wmgal2jy18l3|4cdfdf9a]-HelperThread-#2,5,main]
                            java.net.SocketInputStream.socketRead0(Native Method)
                            java.net.SocketInputStream.read(SocketInputStream.java:152)
                            java.net.SocketInputStream.read(SocketInputStream.java:122)
                            com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
                            com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
                            com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
                            com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
                            com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:598)
                            com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1086)
                            com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
                            com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2344)
                            com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
                            com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                            com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                            sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
                            sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                            java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                            com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
                            com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                            com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                            com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                            com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                            com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                            com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                            com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
            {noformat}
            Danilo Tuler created issue -

              Unassigned Unassigned
              danilo.tuler Danilo Tuler
              Affected customers:
              0 This affects my team
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: