When a db exception is thrown while full index rebuild runs, the indexing completes, but no further incremental indexing jobs are executed.

      It seems that the exception is not handled well, which results in the problems with future indexing jobs.

      The issue is discussed at http://forums.atlassian.com/thread.jspa?threadID=40706&tstart=0&messageID=257331096

      Similar issue were also observed when our server ran our of local disk space on the mount where the index is stored.

            [CONFSERVER-18452] Site indexing stops when a db exception occurs

            Ryan added a comment -

            IndexingTask.runInternal() method is responsible for this:

            public void runInternal() {
                batchIndexer.indexEntities(progress);
                finishedReindexing = true;
            }
            

            If batchIndexer.indexEntries(...) throws an exception then the finishedReindexing flag will never be set to true, causing checks by the DefaultConfluenceIndexManager to always think that indexing is still happening.

            Ryan added a comment - IndexingTask.runInternal() method is responsible for this: public void runInternal() { batchIndexer.indexEntities(progress); finishedReindexing = true ; } If batchIndexer.indexEntries(...) throws an exception then the finishedReindexing flag will never be set to true, causing checks by the DefaultConfluenceIndexManager to always think that indexing is still happening.

            Just for the record, here is a concrete exception that results in this issue. The root cause of this exception is most likely a very different issue, but the side effect is that the indexing stops.

            WARN  2010-02-03 12:33:44,953 [confluence-interactive-reindexing-thread] DefaultConnectionTester:statusOnException - SQL State '08007' of Exception tested by status
            OnException() implies that the database is invalid, and the pool should refill itself with fresh Connections.
             -- referer: http://wikis.sun.com/admin/search-indexes.action | url: /admin/reindex.action | userName: igorminar | action: reindex
            com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution unknown.
                    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
                    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
                    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
                    at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
                    at com.mysql.jdbc.Util.getInstance(Util.java:381)
                    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:985)
                    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957)
                    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
                    at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1561)
                    at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:803)
                    at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:63)
                    at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510)
                    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662)
                    at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632)
                    at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314)
                    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
                    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
                    at $Proxy71.indexEntities(Unknown Source)        at com.atlassian.confluence.search.IndexingTask.runInternal(IndexingTask.java:31)
                    at com.atlassian.confluence.util.longrunning.ConfluenceAbstractLongRunningTask.run(ConfluenceAbstractLongRunningTask.java:21)
                    at java.lang.Thread.run(Thread.java:619)
            WARN  2010-02-03 12:33:44,956 [confluence-interactive-reindexing-thread] C3P0PooledConnectionPool$ConnectionEventListenerImpl:doMarkPoolStatus - A ConnectionTest ha
            s failed, reporting that all previously acquired Connections are likely invalid. The pool will be reset. -- referer: http://wikis.sun.com/admin/search-indexes.action | url: /admin/reindex.action | userName: igorminar | action: reindex
            ERROR 2010-02-03 12:33:44,957 [confluence-interactive-reindexing-thread] JDBCTransaction:commit - Commit failed -- referer: http://wikis.sun.com/admin/search-indexes.action | url: /admin/reindex.action | userName: igorminar | action: reindex
            com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution unknown.
                    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
                    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
                    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
                    at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
                    at com.mysql.jdbc.Util.getInstance(Util.java:381)
                    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:985)
                    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957)
                    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
                    at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1561)
                    at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:803)
                    at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:63)
                    at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510)
                    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662)
                    at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632)
                    at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314)
                    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
                    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
                    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
                    at $Proxy71.indexEntities(Unknown Source)
                    at com.atlassian.confluence.search.IndexingTask.runInternal(IndexingTask.java:31)
                    at com.atlassian.confluence.util.longrunning.ConfluenceAbstractLongRunningTask.run(ConfluenceAbstractLongRunningTask.java:21)
                    at java.lang.Thread.run(Thread.java:619)
            

            Igor Minar added a comment - Just for the record, here is a concrete exception that results in this issue. The root cause of this exception is most likely a very different issue, but the side effect is that the indexing stops. WARN 2010-02-03 12:33:44,953 [confluence-interactive-reindexing-thread] DefaultConnectionTester:statusOnException - SQL State '08007' of Exception tested by status OnException() implies that the database is invalid, and the pool should refill itself with fresh Connections. -- referer: http: //wikis.sun.com/admin/search-indexes.action | url: /admin/reindex.action | userName: igorminar | action: reindex com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution unknown. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at com.mysql.jdbc.Util.handleNewInstance(Util.java:406) at com.mysql.jdbc.Util.getInstance(Util.java:381) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:985) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927) at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1561) at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:803) at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:63) at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510) at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662) at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632) at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy71.indexEntities(Unknown Source) at com.atlassian.confluence.search.IndexingTask.runInternal(IndexingTask.java:31) at com.atlassian.confluence.util.longrunning.ConfluenceAbstractLongRunningTask.run(ConfluenceAbstractLongRunningTask.java:21) at java.lang. Thread .run( Thread .java:619) WARN 2010-02-03 12:33:44,956 [confluence-interactive-reindexing-thread] C3P0PooledConnectionPool$ConnectionEventListenerImpl:doMarkPoolStatus - A ConnectionTest ha s failed, reporting that all previously acquired Connections are likely invalid. The pool will be reset. -- referer: http: //wikis.sun.com/admin/search-indexes.action | url: /admin/reindex.action | userName: igorminar | action: reindex ERROR 2010-02-03 12:33:44,957 [confluence-interactive-reindexing-thread] JDBCTransaction:commit - Commit failed -- referer: http: //wikis.sun.com/admin/search-indexes.action | url: /admin/reindex.action | userName: igorminar | action: reindex com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution unknown. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at com.mysql.jdbc.Util.handleNewInstance(Util.java:406) at com.mysql.jdbc.Util.getInstance(Util.java:381) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:985) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927) at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1561) at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:803) at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:63) at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510) at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662) at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632) at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy71.indexEntities(Unknown Source) at com.atlassian.confluence.search.IndexingTask.runInternal(IndexingTask.java:31) at com.atlassian.confluence.util.longrunning.ConfluenceAbstractLongRunningTask.run(ConfluenceAbstractLongRunningTask.java:21) at java.lang. Thread .run( Thread .java:619)

              rthomas@atlassian.com Ryan
              15d9a6950818 Igor Minar
              Affected customers:
              0 This affects my team
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: