Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-88209

Deadlock in Confluence Questions plugin when two threads try to initialize classes from com.mysema.querydsl library

XMLWordPrintable

      Issue Summary

      This is reproducible on Data Center: yes

      Site restore could stuck after sending ImportFinishedEvent and Confluence Questions plugin tries to rebuild QuestionEdgeIndex.

      Steps to Reproduce

      1. It's a race condition, so it's difficult to reproduce, but such situation could happen during site restore on instance where Confluence Question is installed. This problem is caused by a known issue in the com.mysema.querydsl library

      Expected Results

      Site restore is successful

      Actual Results

      Confluence stuck on the confluence questions plugin indexing:

      "ConfluenceBackupRestoreManager:thread-1" #136446 prio=5 os_prio=0 cpu=2335.99ms elapsed=78667.96s tid=0x00007f030dc03880 nid=0x2e0b in Object.wait()  [0x00007f02e62ef000]
   java.lang.Thread.State: RUNNABLE
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper.getSqlTemplateBuilderByDialect(QueryDslHelper.java:46)
    - waiting on the Class initialization monitor for com.atlassian.confluence.plugins.questions.util.QueryDslHelper$HibernateDialectToSqlTemplateEnum
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper.getConfiguration(QueryDslHelper.java:50)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeQuestionRepository.getRecentQuestionIds(ContentTypeQuestionRepository.java:582)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeQuestionRepository.getRecentQuestions(ContentTypeQuestionRepository.java:547)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeQuestionRepository.getQuestions(ContentTypeQuestionRepository.java:373)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository$DatabaseQueryChainAction.execute(ChainedQueryableQuestionRepository.java:85)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository$DatabaseQueryChainAction.execute(ChainedQueryableQuestionRepository.java:82)
    at com.atlassian.confluence.plugins.questions.util.ChainActionManagerImpl.execute(ChainActionManagerImpl.java:44)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository$LuceneQueryChainAction.execute(ChainedQueryableQuestionRepository.java:55)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository$LuceneQueryChainAction.execute(ChainedQueryableQuestionRepository.java:47)
    at com.atlassian.confluence.plugins.questions.util.ChainActionManagerImpl.execute(ChainActionManagerImpl.java:44)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository.getQuestions(ChainedQueryableQuestionRepository.java:40)
    at com.atlassian.confluence.plugins.questions.service.QuestionServiceImpl.getQuestions(QuestionServiceImpl.java:372)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.7/Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.7/NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
    at com.atlassian.confluence.plugins.questions.internal.security.AuthorizationHandler.invoke(AuthorizationHandler.java:63)
    at jdk.proxy257.$Proxy3682.getQuestions(jdk.proxy257/Unknown Source)
    at com.atlassian.confluence.plugins.questions.service.QuestionIterable$QuestionIterator.hasNext(QuestionIterable.java:54)
    at com.atlassian.confluence.plugins.questions.edgeindex.AbstractQuestionsEdgeIndexBuilder.rebuild(AbstractQuestionsEdgeIndexBuilder.java:48)
    at com.atlassian.confluence.plugins.questions.edgeindex.custom.QuestionsEdgeIndexBuilder.rebuild(QuestionsEdgeIndexBuilder.java:42)
    at com.atlassian.confluence.plugins.questions.internal.listeners.ImportFinishedListener.onImportFinishedEvent(ImportFinishedListener.java:25)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.7/Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.7/NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
    at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:42)
    at com.atlassian.confluence.event.ConfluenceListenerHandlersConfiguration$TimedListenerInvoker.invoke(ConfluenceListenerHandlersConfiguration.java:96)
    at com.atlassian.event.internal.ComparableListenerInvoker.invoke(ComparableListenerInvoker.java:48)
    at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.lambda$null$0(AsynchronousAbleEventDispatcher.java:37)
    at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$1360/0x0000000801ede3e8.run(Unknown Source)
    at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$648/0x00000008019619c0.execute(Unknown Source)
    at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:85)
    at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:114)
    at com.atlassian.confluence.event.TimingEventPublisher.publish(TimingEventPublisher.java:76)
    at com.atlassian.confluence.impl.backuprestore.restore.events.OnRestoreEventsSender.lambda$sendFinishSiteImportEvents$1(OnRestoreEventsSender.java:152)
    at com.atlassian.confluence.impl.backuprestore.restore.events.OnRestoreEventsSender$$Lambda$7339/0x0000000804f1e810.doInTransaction(Unknown Source)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
    at com.atlassian.confluence.impl.backuprestore.restore.dao.RestoreDao.doInTransaction(RestoreDao.java:209)
    at com.atlassian.confluence.impl.backuprestore.restore.dao.RestoreDao.doInTransaction(RestoreDao.java:196)
    at com.atlassian.confluence.impl.backuprestore.restore.events.OnRestoreEventsSender.sendFinishSiteImportEvents(OnRestoreEventsSender.java:151)
    at com.atlassian.confluence.impl.backuprestore.restore.events.OnRestoreEventsSender.sendFinishEvents(OnRestoreEventsSender.java:64)
    at com.atlassian.confluence.impl.backuprestore.restore.RestoreService.doRestore(RestoreService.java:232)
    at com.atlassian.confluence.impl.backuprestore.restore.SiteRestoreService.doRestoreSynchronously(SiteRestoreService.java:34)
    at com.atlassian.confluence.impl.backuprestore.BackupRestoreJobRunner.runJobSynchronously(BackupRestoreJobRunner.java:68)
    at com.atlassian.confluence.impl.backuprestore.ConfluenceBackupRestoreManager.runJobAndUpdateJobProperties(ConfluenceBackupRestoreManager.java:277)
    at com.atlassian.confluence.impl.backuprestore.ConfluenceBackupRestoreManager.processJobsFromTheQueue(ConfluenceBackupRestoreManager.java:259)
    at com.atlassian.confluence.impl.backuprestore.ConfluenceBackupRestoreManager$$Lambda$7100/0x0000000804dce638.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.7/Executors.java:539)
    at java.util.concurrent.FutureTask.run(java.base@17.0.7/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.7/ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.7/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)

      There are also other threads stuck at initializing com.mysema.query.types.Ops and com.mysema.query.types.OperatorImpl:

      "Indexer: 1" #136848 prio=5 os_prio=0 cpu=22.87ms elapsed=78583.52s tid=0x00007f03795bbc90 nid=0x2fc8 in Object.wait()  [0x00007f03087c3000]
   java.lang.Thread.State: RUNNABLE
    at jdk.internal.misc.Unsafe.ensureClassInitialized0(java.base@17.0.7/Native Method)
    - waiting on the Class initialization monitor for com.mysema.query.types.Ops$MathOps
    at jdk.internal.misc.Unsafe.ensureClassInitialized(java.base@17.0.7/Unsafe.java:1155)
    at jdk.internal.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(java.base@17.0.7/UnsafeFieldAccessorFactory.java:42)
    at jdk.internal.reflect.ReflectionFactory.newFieldAccessor(java.base@17.0.7/ReflectionFactory.java:185)
    at java.lang.reflect.Field.acquireFieldAccessor(java.base@17.0.7/Field.java:1132)
    at java.lang.reflect.Field.getFieldAccessor(java.base@17.0.7/Field.java:1113)
    at java.lang.reflect.Field.get(java.base@17.0.7/Field.java:425)
    at com.mysema.query.types.OperatorImpl.<clinit>(OperatorImpl.java:42)
    at com.mysema.query.types.Ops.<clinit>(Ops.java:30)
    at com.mysema.query.types.Templates.<init>(Templates.java:50)
    at com.mysema.query.types.Templates.<init>(Templates.java:38)
    at com.mysema.query.types.Templates.<clinit>(Templates.java:27)
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper$HibernateDialectToSqlTemplateEnum.<clinit>(QueryDslHelper.java:23)
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper.getSqlTemplateBuilderByDialect(QueryDslHelper.java:46)
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper.getConfiguration(QueryDslHelper.java:50)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeAnswerRepository.getNumberOfAnswersByQuestionId(ContentTypeAnswerRepository.java:300)
    ...

      "Indexer: 2" #136849 prio=5 os_prio=0 cpu=10.92ms elapsed=78583.52s tid=0x00007f037a206920 nid=0x2fc9 in Object.wait()  [0x00007f0308ecb000]
   java.lang.Thread.State: RUNNABLE
    at com.mysema.query.types.Ops$MathOps.<clinit>(Ops.java:340)
    - waiting on the Class initialization monitor for com.mysema.query.types.OperatorImpl
    at com.mysema.query.types.expr.NumberExpression.<clinit>(NumberExpression.java:46)
    at com.mysema.query.types.path.BeanPath.createNumber(BeanPath.java:257)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.ContentTable.<init>(ContentTable.java:32)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.QueryDSLMapper$11.apply(QueryDSLMapper.java:63)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.QueryDSLMapper$11.apply(QueryDSLMapper.java:60)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.QueryDSLMapper.getMapping(QueryDSLMapper.java:49)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.QueryDSLMapper.getMapping(QueryDSLMapper.java:40)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeFeaturedTopicRepository.getSQLBindings(ContentTypeFeaturedTopicRepository.java:114)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeFeaturedTopicRepository.getFeaturedTopicIds(ContentTypeFeaturedTopicRepository.java:107)
    ...

      Workaround

      Stop Confluence, revert back to it's original state, upgrade Confluence Questions plugin and start Site restore again

              d5dce7b13926 agawron
              03cb0c04aa4f Irina Tiapchenko
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: