Database connections are not returned to the BoneCP pool when a physical connection is broken

XMLWordPrintable

    • Severity 3 - Minor
    • 1

      Symptoms

      In case when a database connection is terminated (possibly due to network error) before a tracked statement is closed, the database connection may not be returned to the BoneCP pool, resulting in a connection leak. This probably occurs at heavier database load as normally statements finish before the connection is closed.

      A stack trace when hibernate connection fails to release because physical connection is already terminated:

      Caused by: java.sql.SQLRecoverableException: Closed Connection
at oracle.jdbc.driver.PhysicalConnection.needLine(PhysicalConnection.java:3247) [ojdbc8-12.2.0.1.jar:12.2.0.1.0]
...
at com.jolbox.bonecp.StatementHandle.closeStatement(StatementHandle.java:151) [bonecp-0.7.1.RELEASE.jar:0.7.1.RELEASE]
at com.jolbox.bonecp.StatementHandle.close(StatementHandle.java:185) [bonecp-0.7.1.RELEASE.jar:0.7.1.RELEASE]
...
at com.cenqua.crucible.hibernate.trackingstatements.TrackingStatementsConnection.closeTrackedStatements(TrackingStatementsConnection.java:52) [fisheye.jar:?]
at com.cenqua.crucible.hibernate.trackingstatements.TrackingStatementsConnection.close(TrackingStatementsConnection.java:88) [fisheye.jar:?]
at com.cenqua.crucible.hibernate.trackingstatements.TrackingStatementsConnectionProvider.closeConnection(TrackingStatementsConnectionProvider.java:34) [fisheye.jar:?]
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.releaseConnection(AbstractSessionImpl.java:312) [hibernate-core-4.2.13.Final.jar:4.2.13.Final]
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.releaseConnection(LogicalConnectionImpl.java:241) [hibernate-core-4.2.13.Final.jar:4.2.13.Final]

      A typical web thread waiting for a DB connection when all available connections are exhausted:

      "qtp402207819-200" #200 prio=5 os_prio=31 tid=0x00007fb15dfcd000 nid=0x11e0f waiting on condition [0x0000700018785000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007811d11a0> (a com.jolbox.bonecp.BoundedLinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at jsr166y.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:691)
at jsr166y.LinkedTransferQueue.xfer(LinkedTransferQueue.java:600)
at jsr166y.LinkedTransferQueue.poll(LinkedTransferQueue.java:1068)
at com.jolbox.bonecp.BoundedLinkedTransferQueue.poll(BoundedLinkedTransferQueue.java:78)
at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:500) <<<<<<<<<
at com.cenqua.crucible.hibernate.trackingstatements.HibernateBonceCPConnectionProvider.getConnection(HibernateBonceCPConnectionProvider.java:171)
at com.cenqua.crucible.hibernate.trackingstatements.TrackingStatementsConnectionProvider.getConnection(TrackingStatementsConnectionProvider.java:26)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:301)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)

      Application becomes unresponsive as web threads (qtp-*) are hanging on BoneCP.getConnection.

      Steps to reproduce

      T.B.D.

      Workaround

      Restart Fisheye server.

      Related issues

      • FE-7123 upgrade to BoneCP 0.8.0, which contains several bug fixes for leaking connections
      • FE-7127 migration to HikariCP, a successor of BoneCP
      • CRUC-7083 a fix delivered for non-closed statements in 3.8.0

            Assignee:
            Unassigned
            Reporter:
            Marek Parfianowicz
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: