-
Type:
Suggestion
-
Resolution: Unresolved
-
None
-
Component/s: Search - Indexing
-
None
We have upgraded Confluence Datacenter from 8.5.19 to 9.2.13 and faced with issue:
Content Indexing - Queued for processing grows to ~ 40000 - 50000 everyday.
Actual results missed at MY WORK - Recently worked on and ‘Recently Updated’ Macro doesn't shows actual results.
Exception in atlassian-confluence-index.log like this:
WARN [Caesium-1-4] [internal.index.v2.CompositeExtractor] extract com.atlassian.confluence.internal.index.v2.BulkExtractor2Adapter@6e2f2db5 failed to extract fields from Attachment: sheme of logic v.2 (313298403) username.usr: org.hibernate.exception.GenericJDBCException: could not extract ResultSet WARN [Caesium-1-4] [internal.index.v2.CompositeExtractor] extract com.atlassian.confluence.internal.index.v2.BulkExtractor2Adapter@44b0c7da failed to extract fields from Attachment: image-2023-4-12_15-44-36.png v.1 (304474509) username.usr: org.hibernate.exception.GenericJDBCException: could not extract ResultSet WARN [Caesium-1-4] [confluence.impl.search.IndexFlushScheduler] lambda$new$1 Failed to flush index queue LuceneContentIndexFlusher, retry in 600s com.atlassian.confluence.search.v2.SearchIndexAccessException: Unexpected Exception while executing a batched index action against index: [confluence-content] at com.atlassian.confluence.plugins.opensearch.OpenSearchSearchIndexAccessor.withBatchUpdate(OpenSearchSearchIndexAccessor.java:120) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at com.atlassian.confluence.plugins.opensearch.spring.interceptors.OpenSearchExceptionHandlingInterceptor.invoke(OpenSearchExceptionHandlingInterceptor.java:49) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) at jdk.proxy111/jdk.proxy111.$Proxy2233.withBatchUpdate(Unknown Source) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.atlassian.confluence.impl.osgi.OsgiProxyFactory$ServiceTrackingProxy.invoke(OsgiProxyFactory.java:75) at jdk.proxy4/jdk.proxy4.$Proxy260.withBatchUpdate(Unknown Source) at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.performFlush(LuceneIncrementalIndexManager.java:173) at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.flushQueue(LuceneIncrementalIndexManager.java:125) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223) at jdk.proxy4/jdk.proxy4.$Proxy267.flushQueue(Unknown Source) at com.atlassian.confluence.impl.search.IndexFlushScheduler.lambda$new$1(IndexFlushScheduler.java:118) at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.doRunJob(JobRunnerWrapper.java:121) at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:83) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$JobRunnerWithStats.runJob(CaesiumSchedulerService.java:787) at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:145) at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:117) at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:99) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:578) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJobWithRetryOnFailure(CaesiumSchedulerService.java:533) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:461) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35) at java.base/java.lang.Thread.run(Unknown Source) Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not extract ResultSet at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1626) at com.atlassian.confluence.impl.journal.HibernateJournalDao.findEntries(HibernateJournalDao.java:69) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:14) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223) at jdk.proxy4/jdk.proxy4.$Proxy190.findEntries(Unknown Source) at com.atlassian.confluence.impl.journal.DefaultJournalManager.processNewEntries(DefaultJournalManager.java:116) at com.atlassian.confluence.impl.journal.DefaultJournalService.processNewEntries(DefaultJournalService.java:44) at com.atlassian.confluence.impl.index.DefaultIndexJournalService.processNewEntries(DefaultIndexJournalService.java:52) at com.atlassian.confluence.internal.search.queue.AbstractJournalIndexTaskQueue.flushAndExecute(AbstractJournalIndexTaskQueue.java:185) at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.lambda$flushAndExecuteInBatches$3(LuceneIncrementalIndexManager.java:232) at com.atlassian.confluence.impl.hibernate.HibernateSessionManager5.executeThenClearSessionWithoutCommitOrFlush(HibernateSessionManager5.java:167) at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.flushAndExecuteInBatches(LuceneIncrementalIndexManager.java:229) at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.lambda$performFlush$1(LuceneIncrementalIndexManager.java:183) at com.atlassian.confluence.plugins.opensearch.OpenSearchSearchIndexAccessor.execute(OpenSearchSearchIndexAccessor.java:78) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at com.atlassian.confluence.plugins.opensearch.spring.interceptors.OpenSearchExceptionHandlingInterceptor.invoke(OpenSearchExceptionHandlingInterceptor.java:49) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) at jdk.proxy111/jdk.proxy111.$Proxy2233.execute(Unknown Source) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.atlassian.confluence.impl.osgi.OsgiProxyFactory$ServiceTrackingProxy.invoke(OsgiProxyFactory.java:75) at jdk.proxy4/jdk.proxy4.$Proxy260.execute(Unknown Source) at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.lambda$performFlush$2(LuceneIncrementalIndexManager.java:173) at com.atlassian.confluence.plugins.opensearch.OpenSearchSearchIndexAccessor.withBatchUpdate(OpenSearchSearchIndexAccessor.java:118) ... 39 more Caused by: org.hibernate.exception.GenericJDBCException: could not extract ResultSet at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:67) at org.hibernate.loader.Loader.getResultSet(Loader.java:2322) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2075) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2037) at org.hibernate.loader.Loader.doQuery(Loader.java:956) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:357) at org.hibernate.loader.Loader.doList(Loader.java:2868) at org.hibernate.loader.Loader.doList(Loader.java:2850) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2682) at org.hibernate.loader.Loader.list(Loader.java:2677) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:540) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:218) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1459) at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1649) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617) ... 76 more Caused by: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:137) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57) ... 91 more Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute nextval() in a read-only transaction at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:137) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) at net.java.ao.ForwardingPreparedStatement.executeQuery(ForwardingPreparedStatement.java:40) at net.java.ao.ParameterMetadataCachingPreparedStatement.executeQuery(ParameterMetadataCachingPreparedStatement.java:10) at net.java.ao.db.PostgreSQLDatabaseProvider.insertReturningKey(PostgreSQLDatabaseProvider.java:306) at net.java.ao.EntityManager.create(EntityManager.java:413) at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.create(EntityManagedActiveObjects.java:94) at com.atlassian.activeobjects.osgi.ActiveObjectsDelegate.create(ActiveObjectsDelegate.java:241) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) at jdk.proxy233/jdk.proxy233.$Proxy3572.create(Unknown Source) at com.requirementyogi.datacenter.sharedservices.queue.managers.QueueManager.addToQueue(QueueManager.java:62) at com.requirementyogi.datacenter.sharedservices.queue.managers.QueueAPIImpl.addToQueue(QueueAPIImpl.java:79) at com.requirementyogi.datacenter.confluenceapp.indexing.IndexingService.enqueueIndexing(IndexingService.java:124) at com.requirementyogi.datacenter.confluenceapp.indexing.ProxyIndexingService.fastResetExcerpts(ProxyIndexingService.java:102) at com.requirementyogi.datacenter.confluenceapp.indexing.PageEvents.onAttachmentUpdatedEvent(PageEvents.java:161) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:42) at com.atlassian.confluence.event.DelegatingListenerInvoker.invoke(DelegatingListenerInvoker.java:23) at com.atlassian.confluence.event.ConfluenceListenerInvoker.invoke(ConfluenceListenerInvoker.java:30) at com.atlassian.event.internal.ComparableListenerInvoker.invoke(ComparableListenerInvoker.java:48) at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.lambda$static$0(AsynchronousAbleEventDispatcher.java:38) at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:89) at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:103) at com.atlassian.confluence.event.TimingEventPublisher.publish(TimingEventPublisher.java:75) at com.atlassian.confluence.impl.search.v2.extractor.AttachmentExtractedTextExtractor.extractText(AttachmentExtractedTextExtractor.java:167) at com.atlassian.confluence.internal.index.v2.CompositeExtractor.extract(CompositeExtractor.java:53) at com.atlassian.confluence.internal.index.v2.Extractor2DocumentBuilder.build(Extractor2DocumentBuilder.java:48) at com.atlassian.confluence.internal.index.v2.AtlassianContentDocumentBuilder.build(AtlassianContentDocumentBuilder.java:67) at com.atlassian.confluence.internal.index.v2.AtlassianContentDocumentBuilder.build(AtlassianContentDocumentBuilder.java:34) at com.atlassian.confluence.internal.index.v2.MergingAtlassianDocumentBuilder.lambda$buildDocuments$0(MergingAtlassianDocumentBuilder.java:38) at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) at com.atlassian.confluence.internal.index.v2.MergingAtlassianDocumentBuilder.buildDocuments(MergingAtlassianDocumentBuilder.java:38) at com.atlassian.confluence.internal.index.v2.MergingAtlassianDocumentBuilder.build(MergingAtlassianDocumentBuilder.java:31) at com.atlassian.confluence.internal.search.tasks.AddDocumentIndexTask.perform(AddDocumentIndexTask.java:56) at com.atlassian.confluence.internal.search.tasks.UpdateDocumentIndexTask.perform(UpdateDocumentIndexTask.java:90) at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.lambda$performFlush$0(LuceneIncrementalIndexManager.java:177) at com.atlassian.confluence.internal.search.queue.AbstractJournalIndexTaskQueue.lambda$flushAndExecute$1(AbstractJournalIndexTaskQueue.java:195) at com.atlassian.confluence.impl.journal.DefaultJournalService.lambda$convert$0(DefaultJournalService.java:87) at com.atlassian.confluence.impl.journal.DefaultJournalManager.processNewEntries(DefaultJournalManager.java:117) ... 62 more
A full index rebuild goes through without errors, but then the queue starts growing again.
We changed Lucene to OpenSearch but that didn't resolve the issue.