Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-59303

Race condition in entity engine connection handling

    XMLWordPrintable

    Details

      Description

      There is a race condition in Atlassian's OfBiz Entity Engine fork that can result in connections being detected incorrectly as abandoned when they are actually still in use. This is due to certain allowed changes to object visibility as a result of just-in-time compiler optimizations, with the result that the connection is enqueued as abandoned just before the thread that is using it closes it.

      The most common manifestation of this is a warning message about an abandoned SQL processor:

      !!! ABANDONED SQLProcessor DETECTED !!!
      	This probably means that somebody forgot to close an EntityListIterator.
      	Connection: org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker$DelegatingConnectionImpl@13af30e
      

      This is particularly likely under heavy use (such as when reindexing) and when the connection is opened and closed in a small function that is likely to inline these operations, such as this from JCHART (leading to JCHART-462 as a result):

              try
              {
                  sqlProcessor = new SQLProcessor(ENTITY_DS);
              }
              finally
              {
      ...
                          sqlProcessor.close();
      ...
              }
      ...
      
      

      In the most extreme cases (typically under heavy load) it is even possible for the connection to be closed by another thread before its connection commits, with actual failed operations as a result:

      Exception in thread "TestThread[59]"
      org.ofbiz.core.entity.GenericDataSourceException: SQL Exception occurred on commit (Connection org.hsqldb.jdbc.jdbcConnection@d1b458d is closed.)
      	at org.ofbiz.core.entity.jdbc.SQLProcessor.commit(SQLProcessor.java:211)
      	at org.ofbiz.core.entity.jdbc.SQLProcessor.smartCommit(SQLProcessor.java:362)
      	at org.ofbiz.core.entity.jdbc.SQLProcessor.close(SQLProcessor.java:264)
      ...
      Caused by: java.sql.SQLException: Connection org.hsqldb.jdbc.jdbcConnection@d1b458d is closed.
      	at org.apache.commons.dbcp2.DelegatingConnection.checkOpen(DelegatingConnection.java:605)
      	at org.apache.commons.dbcp2.DelegatingConnection.commit(DelegatingConnection.java:362)
      	at org.apache.commons.dbcp2.DelegatingConnection.commit(DelegatingConnection.java:364)
      	at org.ofbiz.core.entity.jdbc.interceptors.connection.DelegatingConnection.commit(DelegatingConnection.java:66)
      	at org.ofbiz.core.entity.jdbc.SQLProcessor.commit(SQLProcessor.java:205)
      ...
      

      This is why using a ReadOnlySQLProcessor works around the problem; the read-only connections do not attempt to commit, so the fact that it was already closed by another thread makes no difference.

      For the technically savvy, there is a reasonably good and detailed explanation of this rather subtle problem in this blog post. The fix is to reorder the volatile writes and the release of the connection guard in such a way as to disallow the visibility change.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              ohernandez@atlassian.com Oswaldo Hernandez (Inactive)
              Reporter:
              cfuller crf
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: