Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-4453

Hibernate Session OutOfMemoryError AND ORA-01000

    XMLWordPrintable

Details

    Description

      A bug in hibernate 4.3.0 shipped within Stash 2.11.2 could potentially cause two issues with both Oracle and MySQL databases.

      These issues are now fixed on Stash 2.11.3, which ships with Hibernate 4.3.4.

      If you are on 2.11.2 with and hitting any "Out of Memory" log messages similar to the ones below, please upgrade your instance to 2.11.3 as you are most likely hitting this issue. Oracle DB users affected by this issue may see ORA-01000 error messages.

      Example MySQL log entries in atlassian-stash.log:

      The below exceptions are examples. Any error messages that mention "java.lang.OutOfMemoryError" on Stash 2.11.2 may be caused by this bug.

      2014-03-03 00:22:36,416 ERROR [http-bio-7990-exec-33] 22x19473x8 10.7.30.48 "GET /mvc/error500 HTTP/1.1" c.a.s.i.web.ErrorPageController There was an unhandled exception loading [/scm/bldtls/dev-ci-scripts.git/info/refs]
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is java.lang.OutOfMemoryError: Java heap space
	at org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:517) ~[spring-orm-4.0.0.RELEASE.jar:4.0.0.RELEASE]
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-4.0.0.RELEASE.jar:4.0.0.RELEASE]
	at com.sun.proxy.$Proxy151.getStatus(Unknown Source) ~[na:na]
	at com.atlassian.stash.internal.check.LicensingCheck.check(LicensingCheck.java:36) ~[stash-service-impl-2.11.2.jar:na]
	at com.atlassian.stash.internal.check.ScmRequestCheckServiceImpl.checkActionAllowed(ScmRequestCheckServiceImpl.java:50) ~[stash-service-impl-2.11.2.jar:na]
	at com.sun.proxy.$Proxy158.checkActionAllowed(Unknown Source) ~[na:na]
	at com.atlassian.stash.internal.scm.ScmServlet.handleRequest(ScmServlet.java:51) ~[ScmServlet.class:na]
	at org.springframework.web.context.support.HttpRequestHandlerServlet.service(HttpRequestHandlerServlet.java:68) ~[spring-web-4.0.0.RELEASE.jar:4.0.0.RELEASE]
	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:25) ~[na:na]
	at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:118) ~[StashAuthenticationFilter.class:na]
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:111) ~[BeforeLoginPluginAuthenticationFilter.class:na]
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:77) ~[BeforeLoginPluginAuthenticationFilter.class:na]
	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:100) ~[atlassian-trusted-apps-core-3.0.3.jar:na]
	at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:61) ~[na:na]
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:89) ~[BeforeLoginPluginAuthenticationFilter.class:na]
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75) ~[BeforeLoginPluginAuthenticationFilter.class:na]
	at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:73) ~[stash-service-impl-2.11.2.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
	at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
	... 187 frames trimmed
java.lang.OutOfMemoryError: Java heap space

      Oracle log entries in atlassian-stash.log

      2014-02-27 12:30:43,765 WARN  [scheduler_Worker-6]  o.h.e.jdbc.spi.SqlExceptionHelper SQL Error: 1000, SQLState: 72000
2014-02-27 12:30:43,765 ERROR [scheduler_Worker-6]  o.h.e.jdbc.spi.SqlExceptionHelper ORA-01000: maximum open cursors exceeded
2014-02-27 12:30:43,841 ERROR [scheduler_Worker-6]  c.a.c.d.DbCachingDirectoryPoller Error occurred while refreshing the cache for directory [ 458754 ].
org.hibernate.exception.GenericJDBCException: could not extract ResultSet
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:89) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.getResultSet(AbstractLoadPlanBasedLoader.java:448) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeQueryStatement(AbstractLoadPlanBasedLoader.java:201) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:137) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:102) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.collection.plan.AbstractLoadPlanBasedCollectionInitializer.initialize(AbstractLoadPlanBasedCollectionInitializer.java:102) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:693) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:92) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:1893) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.collection.internal.AbstractPersistentCollection.forceInitialization(AbstractPersistentCollection.java:668) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.Hibernate.initialize(Hibernate.java:78) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at com.atlassian.stash.internal.crowd.HibernateUserDao.initialize(HibernateUserDao.java:272) ~[stash-dao-impl-2.11.2.jar:na]
	at com.atlassian.stash.internal.crowd.HibernateUserDao.findByName(HibernateUserDao.java:101) ~[stash-dao-impl-2.11.2.jar:na]
	at com.atlassian.stash.internal.crowd.HibernateUserDao.findByName(HibernateUserDao.java:51) ~[stash-dao-impl-2.11.2.jar:na]
	at com.sun.proxy.$Proxy111.findByName(Unknown Source) ~[na:na]
	at com.atlassian.crowd.directory.AbstractInternalDirectory.findUserByName(AbstractInternalDirectory.java:150) ~[crowd-persistence-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.findUserByName(DbCachingRemoteDirectory.java:137) ~[crowd-core-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.findUserByName(DirectoryManagerGeneric.java:298) ~[crowd-core-2.7.2-rc2.jar:na]
	at com.sun.proxy.$Proxy124.findUserByName(Unknown Source) ~[na:na]
	at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.findUserByName(ApplicationServiceGeneric.java:299) ~[crowd-core-2.7.2-rc2.jar:na]
	at com.sun.proxy.$Proxy129.findUserByName(Unknown Source) ~[na:na]
	at com.atlassian.crowd.embedded.core.CrowdServiceImpl.getUser(CrowdServiceImpl.java:93) ~[embedded-crowd-core-2.7.2-rc2.jar:na]
	at com.sun.proxy.$Proxy130.getUser(Unknown Source) ~[na:na]
	at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.getCrowdUser(CrowdAwareLoadListener.java:121) ~[stash-dao-impl-2.11.2.jar:na]
	at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.setCrowdUserIfNecessary(CrowdAwareLoadListener.java:134) ~[stash-dao-impl-2.11.2.jar:na]
	at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.onPostLoad(CrowdAwareLoadListener.java:58) ~[stash-dao-impl-2.11.2.jar:na]
	at org.hibernate.engine.internal.TwoPhaseLoad.postLoad(TwoPhaseLoad.java:326) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1136) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.Loader.processResultSet(Loader.java:972) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.Loader.doQuery(Loader.java:920) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:354) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.Loader.doList(Loader.java:2551) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.Loader.doList(Loader.java:2537) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2367) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.Loader.list(Loader.java:2362) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:496) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:387) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:229) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1260) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.internal.QueryImpl.list(QueryImpl.java:103) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at org.hibernate.internal.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:966) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	at com.atlassian.stash.internal.user.HibernateStashUserDao.findByName(HibernateStashUserDao.java:123) ~[stash-dao-impl-2.11.2.jar:na]
	at com.sun.proxy.$Proxy109.findByName(Unknown Source) ~[na:na]
	at com.atlassian.stash.internal.crowd.HibernateUserDao.createIfNeeded(HibernateUserDao.java:230) ~[stash-dao-impl-2.11.2.jar:na]
	at com.atlassian.stash.internal.crowd.HibernateUserDao.addAll(HibernateUserDao.java:85) ~[stash-dao-impl-2.11.2.jar:na]
	at com.sun.proxy.$Proxy111.addAll(Unknown Source) ~[na:na]
	at com.atlassian.crowd.directory.CachingDirectory.addAllUsers(CachingDirectory.java:97) ~[crowd-persistence-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.directory.DbCachingRemoteChangeOperations.addUsers(DbCachingRemoteChangeOperations.java:181) ~[crowd-core-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.directory.$Proxy1364.addUsers(Unknown Source) ~[na:na]
	at com.atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations.addOrUpdateCachedUsers(DirectoryCacheImplUsingChangeOperations.java:58) ~[crowd-core-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.directory.ldap.cache.UsnChangedCacheRefresher.synchroniseAllUsers(UsnChangedCacheRefresher.java:157) ~[crowd-ldap-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:78) ~[crowd-ldap-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.directory.ldap.cache.UsnChangedCacheRefresher.synchroniseAll(UsnChangedCacheRefresher.java:126) ~[crowd-ldap-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1008) ~[crowd-core-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:75) ~[crowd-core-2.7.2-rc2.jar:na]
	at com.sun.proxy.$Proxy121.synchronise(Unknown Source) ~[na:na]
	at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50) ~[crowd-core-2.7.2-rc2.jar:na]
	at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobBean.executeInternal(DirectoryPollerJobBean.java:29) [crowd-core-2.7.2-rc2.jar:na]
	at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:111) [spring-context-support-4.0.0.RELEASE.jar:4.0.0.RELEASE]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:223) [quartz-1.8.6.jar:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549) [quartz-1.8.6.jar:na]
	... 110 frames trimmed
Caused by: java.sql.SQLException: ORA-01000: maximum open cursors exceeded

	at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:863) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
	at com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:172) ~[bonecp-0.7.1.RELEASE.jar:0.7.1.RELEASE]
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:80) ~[hibernate-core-4.3.0.Final.jar:4.3.0.Final]
	... 61 common frames omitted

      OutOfMemoryError in catalina.out

      Mar 3, 2014 7:48:09 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.8-atlassian-12 07/11/2013 01:08 AM'
Exception in thread "http-bio-7990-exec-1223" java.lang.OutOfMemoryError: Java heap space
Exception in thread "process reaper" Exception in thread "pool-10-thread-3002" 	at org.apache.tomcat.util.buf.ByteChunk.allocate(ByteChunk.java:168)
	at org.apache.tomcat.util.buf.ByteChunk.<init>(ByteChunk.java:130)
	at org.apache.catalina.connector.InputBuffer.<init>(InputBuffer.java:158)
	at org.apache.catalina.connector.InputBuffer.<init>(InputBuffer.java:145)
	at org.apache.catalina.connector.Request.<init>(Request.java:251)
	at org.apache.catalina.connector.Connector.createRequest(Connector.java:885)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space

      Attachments

        Issue Links

          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Activity

            People

              bturner Bryan Turner (Inactive)
              cszmajda Cristan Szmajda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: