Failed to flush index queue LuceneContentIndexFlusher

XMLWordPrintable

    • Type: Suggestion
    • Resolution: Unresolved
    • None
    • Component/s: Search - Indexing
    • None

      We have upgraded Confluence Datacenter from 8.5.19 to 9.2.13 and faced with issue:

      Content Indexing - Queued for processing grows to ~ 40000 - 50000 everyday.

      Actual results missed at MY WORK - Recently worked on and ‘Recently Updated’ Macro doesn't shows actual results.

      Exception in atlassian-confluence-index.log like this:

      WARN [Caesium-1-4] [internal.index.v2.CompositeExtractor] extract com.atlassian.confluence.internal.index.v2.BulkExtractor2Adapter@6e2f2db5 failed to extract fields from Attachment: sheme of logic v.2 (313298403) username.usr: org.hibernate.exception.GenericJDBCException: could not extract ResultSet
      
      WARN [Caesium-1-4] [internal.index.v2.CompositeExtractor] extract com.atlassian.confluence.internal.index.v2.BulkExtractor2Adapter@44b0c7da failed to extract fields from Attachment: image-2023-4-12_15-44-36.png v.1 (304474509) username.usr: org.hibernate.exception.GenericJDBCException: could not extract ResultSet
      
      WARN [Caesium-1-4] [confluence.impl.search.IndexFlushScheduler] lambda$new$1 Failed to flush index queue LuceneContentIndexFlusher, retry in 600s
      com.atlassian.confluence.search.v2.SearchIndexAccessException: Unexpected Exception while executing a batched index action against index: [confluence-content]
              at com.atlassian.confluence.plugins.opensearch.OpenSearchSearchIndexAccessor.withBatchUpdate(OpenSearchSearchIndexAccessor.java:120)
              at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
              at com.atlassian.confluence.plugins.opensearch.spring.interceptors.OpenSearchExceptionHandlingInterceptor.invoke(OpenSearchExceptionHandlingInterceptor.java:49)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
              at jdk.proxy111/jdk.proxy111.$Proxy2233.withBatchUpdate(Unknown Source)
              at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at com.atlassian.confluence.impl.osgi.OsgiProxyFactory$ServiceTrackingProxy.invoke(OsgiProxyFactory.java:75)
              at jdk.proxy4/jdk.proxy4.$Proxy260.withBatchUpdate(Unknown Source)
              at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.performFlush(LuceneIncrementalIndexManager.java:173)
              at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.flushQueue(LuceneIncrementalIndexManager.java:125)
              at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379)
              at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
              at jdk.proxy4/jdk.proxy4.$Proxy267.flushQueue(Unknown Source)
              at com.atlassian.confluence.impl.search.IndexFlushScheduler.lambda$new$1(IndexFlushScheduler.java:118)
              at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.doRunJob(JobRunnerWrapper.java:121)
              at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:83)
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$JobRunnerWithStats.runJob(CaesiumSchedulerService.java:787)
              at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:145)
              at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:117)
              at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:99)
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:578)
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJobWithRetryOnFailure(CaesiumSchedulerService.java:533)
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:461)
              at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
              at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
              at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
              at java.base/java.lang.Thread.run(Unknown Source)
      Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not extract ResultSet
              at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
              at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1626)
              at com.atlassian.confluence.impl.journal.HibernateJournalDao.findEntries(HibernateJournalDao.java:69)
              at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
              at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:14)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379)
              at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
              at jdk.proxy4/jdk.proxy4.$Proxy190.findEntries(Unknown Source)
              at com.atlassian.confluence.impl.journal.DefaultJournalManager.processNewEntries(DefaultJournalManager.java:116)
              at com.atlassian.confluence.impl.journal.DefaultJournalService.processNewEntries(DefaultJournalService.java:44)
              at com.atlassian.confluence.impl.index.DefaultIndexJournalService.processNewEntries(DefaultIndexJournalService.java:52)
              at com.atlassian.confluence.internal.search.queue.AbstractJournalIndexTaskQueue.flushAndExecute(AbstractJournalIndexTaskQueue.java:185)
              at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.lambda$flushAndExecuteInBatches$3(LuceneIncrementalIndexManager.java:232)
              at com.atlassian.confluence.impl.hibernate.HibernateSessionManager5.executeThenClearSessionWithoutCommitOrFlush(HibernateSessionManager5.java:167)
              at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.flushAndExecuteInBatches(LuceneIncrementalIndexManager.java:229)
              at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.lambda$performFlush$1(LuceneIncrementalIndexManager.java:183)
              at com.atlassian.confluence.plugins.opensearch.OpenSearchSearchIndexAccessor.execute(OpenSearchSearchIndexAccessor.java:78)
              at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
              at com.atlassian.confluence.plugins.opensearch.spring.interceptors.OpenSearchExceptionHandlingInterceptor.invoke(OpenSearchExceptionHandlingInterceptor.java:49)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
              at jdk.proxy111/jdk.proxy111.$Proxy2233.execute(Unknown Source)
              at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at com.atlassian.confluence.impl.osgi.OsgiProxyFactory$ServiceTrackingProxy.invoke(OsgiProxyFactory.java:75)
              at jdk.proxy4/jdk.proxy4.$Proxy260.execute(Unknown Source)
              at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.lambda$performFlush$2(LuceneIncrementalIndexManager.java:173)
              at com.atlassian.confluence.plugins.opensearch.OpenSearchSearchIndexAccessor.withBatchUpdate(OpenSearchSearchIndexAccessor.java:118)
              ... 39 more
      Caused by: org.hibernate.exception.GenericJDBCException: could not extract ResultSet
              at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
              at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
              at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
              at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:67)
              at org.hibernate.loader.Loader.getResultSet(Loader.java:2322)
              at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2075)
              at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2037)
              at org.hibernate.loader.Loader.doQuery(Loader.java:956)
              at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:357)
              at org.hibernate.loader.Loader.doList(Loader.java:2868)
              at org.hibernate.loader.Loader.doList(Loader.java:2850)
              at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2682)
              at org.hibernate.loader.Loader.list(Loader.java:2677)
              at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:540)
              at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
              at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:218)
              at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1459)
              at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1649)
              at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617)
              ... 76 more
      Caused by: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
              at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725)
              at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412)
              at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)
              at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)
              at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)
              at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)
              at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:137)
              at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
              at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
              at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57)
              ... 91 more
      Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute nextval() in a read-only transaction
              at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725)
              at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412)
              at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)
              at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)
              at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)
              at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)
              at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:137)
              at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
              at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
              at net.java.ao.ForwardingPreparedStatement.executeQuery(ForwardingPreparedStatement.java:40)
              at net.java.ao.ParameterMetadataCachingPreparedStatement.executeQuery(ParameterMetadataCachingPreparedStatement.java:10)
              at net.java.ao.db.PostgreSQLDatabaseProvider.insertReturningKey(PostgreSQLDatabaseProvider.java:306)
              at net.java.ao.EntityManager.create(EntityManager.java:413)
              at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.create(EntityManagedActiveObjects.java:94)
              at com.atlassian.activeobjects.osgi.ActiveObjectsDelegate.create(ActiveObjectsDelegate.java:241)
              at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
              at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
              at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
              at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
              at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
              at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
              at jdk.proxy233/jdk.proxy233.$Proxy3572.create(Unknown Source)
              at com.requirementyogi.datacenter.sharedservices.queue.managers.QueueManager.addToQueue(QueueManager.java:62)
              at com.requirementyogi.datacenter.sharedservices.queue.managers.QueueAPIImpl.addToQueue(QueueAPIImpl.java:79)
              at com.requirementyogi.datacenter.confluenceapp.indexing.IndexingService.enqueueIndexing(IndexingService.java:124)
              at com.requirementyogi.datacenter.confluenceapp.indexing.ProxyIndexingService.fastResetExcerpts(ProxyIndexingService.java:102)
              at com.requirementyogi.datacenter.confluenceapp.indexing.PageEvents.onAttachmentUpdatedEvent(PageEvents.java:161)
              at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:42)
              at com.atlassian.confluence.event.DelegatingListenerInvoker.invoke(DelegatingListenerInvoker.java:23)
              at com.atlassian.confluence.event.ConfluenceListenerInvoker.invoke(ConfluenceListenerInvoker.java:30)
              at com.atlassian.event.internal.ComparableListenerInvoker.invoke(ComparableListenerInvoker.java:48)
              at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.lambda$static$0(AsynchronousAbleEventDispatcher.java:38)
              at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:89)
              at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:103)
              at com.atlassian.confluence.event.TimingEventPublisher.publish(TimingEventPublisher.java:75)
              at com.atlassian.confluence.impl.search.v2.extractor.AttachmentExtractedTextExtractor.extractText(AttachmentExtractedTextExtractor.java:167)
              at com.atlassian.confluence.internal.index.v2.CompositeExtractor.extract(CompositeExtractor.java:53)
              at com.atlassian.confluence.internal.index.v2.Extractor2DocumentBuilder.build(Extractor2DocumentBuilder.java:48)
              at com.atlassian.confluence.internal.index.v2.AtlassianContentDocumentBuilder.build(AtlassianContentDocumentBuilder.java:67)
              at com.atlassian.confluence.internal.index.v2.AtlassianContentDocumentBuilder.build(AtlassianContentDocumentBuilder.java:34)
              at com.atlassian.confluence.internal.index.v2.MergingAtlassianDocumentBuilder.lambda$buildDocuments$0(MergingAtlassianDocumentBuilder.java:38)
              at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
              at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
              at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
              at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
              at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
              at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
              at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
              at com.atlassian.confluence.internal.index.v2.MergingAtlassianDocumentBuilder.buildDocuments(MergingAtlassianDocumentBuilder.java:38)
              at com.atlassian.confluence.internal.index.v2.MergingAtlassianDocumentBuilder.build(MergingAtlassianDocumentBuilder.java:31)
              at com.atlassian.confluence.internal.search.tasks.AddDocumentIndexTask.perform(AddDocumentIndexTask.java:56)
              at com.atlassian.confluence.internal.search.tasks.UpdateDocumentIndexTask.perform(UpdateDocumentIndexTask.java:90)
              at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.lambda$performFlush$0(LuceneIncrementalIndexManager.java:177)
              at com.atlassian.confluence.internal.search.queue.AbstractJournalIndexTaskQueue.lambda$flushAndExecute$1(AbstractJournalIndexTaskQueue.java:195)
              at com.atlassian.confluence.impl.journal.DefaultJournalService.lambda$convert$0(DefaultJournalService.java:87)
              at com.atlassian.confluence.impl.journal.DefaultJournalManager.processNewEntries(DefaultJournalManager.java:117)
              ... 62 more

      A full index rebuild goes through without errors, but then the queue starts growing again.

      We changed Lucene to OpenSearch but that didn't resolve the issue. 
       

              Assignee:
              Unassigned
              Reporter:
              sergey-kuznetsov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: