Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-88209

Deadlock in Confluence Questions plugin when two threads try to initialize classes from com.mysema.querydsl library

    XMLWordPrintable

Details

    Description

      Issue Summary

      This is reproducible on Data Center: yes

      Site restore could stuck after sending ImportFinishedEvent and Confluence Questions plugin tries to rebuild QuestionEdgeIndex.

      Steps to Reproduce

      1. It's a race condition, so it's difficult to reproduce, but such situation could happen during site restore on instance where Confluence Question is installed. This problem is caused by a known issue in the com.mysema.querydsl library

      Expected Results

      Site restore is successful

      Actual Results

      Confluence stuck on the confluence questions plugin indexing:

      "ConfluenceBackupRestoreManager:thread-1" #136446 prio=5 os_prio=0 cpu=2335.99ms elapsed=78667.96s tid=0x00007f030dc03880 nid=0x2e0b in Object.wait()  [0x00007f02e62ef000]
   java.lang.Thread.State: RUNNABLE
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper.getSqlTemplateBuilderByDialect(QueryDslHelper.java:46)
    - waiting on the Class initialization monitor for com.atlassian.confluence.plugins.questions.util.QueryDslHelper$HibernateDialectToSqlTemplateEnum
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper.getConfiguration(QueryDslHelper.java:50)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeQuestionRepository.getRecentQuestionIds(ContentTypeQuestionRepository.java:582)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeQuestionRepository.getRecentQuestions(ContentTypeQuestionRepository.java:547)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeQuestionRepository.getQuestions(ContentTypeQuestionRepository.java:373)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository$DatabaseQueryChainAction.execute(ChainedQueryableQuestionRepository.java:85)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository$DatabaseQueryChainAction.execute(ChainedQueryableQuestionRepository.java:82)
    at com.atlassian.confluence.plugins.questions.util.ChainActionManagerImpl.execute(ChainActionManagerImpl.java:44)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository$LuceneQueryChainAction.execute(ChainedQueryableQuestionRepository.java:55)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository$LuceneQueryChainAction.execute(ChainedQueryableQuestionRepository.java:47)
    at com.atlassian.confluence.plugins.questions.util.ChainActionManagerImpl.execute(ChainActionManagerImpl.java:44)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ChainedQueryableQuestionRepository.getQuestions(ChainedQueryableQuestionRepository.java:40)
    at com.atlassian.confluence.plugins.questions.service.QuestionServiceImpl.getQuestions(QuestionServiceImpl.java:372)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.7/Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.7/NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
    at com.atlassian.confluence.plugins.questions.internal.security.AuthorizationHandler.invoke(AuthorizationHandler.java:63)
    at jdk.proxy257.$Proxy3682.getQuestions(jdk.proxy257/Unknown Source)
    at com.atlassian.confluence.plugins.questions.service.QuestionIterable$QuestionIterator.hasNext(QuestionIterable.java:54)
    at com.atlassian.confluence.plugins.questions.edgeindex.AbstractQuestionsEdgeIndexBuilder.rebuild(AbstractQuestionsEdgeIndexBuilder.java:48)
    at com.atlassian.confluence.plugins.questions.edgeindex.custom.QuestionsEdgeIndexBuilder.rebuild(QuestionsEdgeIndexBuilder.java:42)
    at com.atlassian.confluence.plugins.questions.internal.listeners.ImportFinishedListener.onImportFinishedEvent(ImportFinishedListener.java:25)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.7/Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.7/NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
    at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:42)
    at com.atlassian.confluence.event.ConfluenceListenerHandlersConfiguration$TimedListenerInvoker.invoke(ConfluenceListenerHandlersConfiguration.java:96)
    at com.atlassian.event.internal.ComparableListenerInvoker.invoke(ComparableListenerInvoker.java:48)
    at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.lambda$null$0(AsynchronousAbleEventDispatcher.java:37)
    at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$1360/0x0000000801ede3e8.run(Unknown Source)
    at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$648/0x00000008019619c0.execute(Unknown Source)
    at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:85)
    at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:114)
    at com.atlassian.confluence.event.TimingEventPublisher.publish(TimingEventPublisher.java:76)
    at com.atlassian.confluence.impl.backuprestore.restore.events.OnRestoreEventsSender.lambda$sendFinishSiteImportEvents$1(OnRestoreEventsSender.java:152)
    at com.atlassian.confluence.impl.backuprestore.restore.events.OnRestoreEventsSender$$Lambda$7339/0x0000000804f1e810.doInTransaction(Unknown Source)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
    at com.atlassian.confluence.impl.backuprestore.restore.dao.RestoreDao.doInTransaction(RestoreDao.java:209)
    at com.atlassian.confluence.impl.backuprestore.restore.dao.RestoreDao.doInTransaction(RestoreDao.java:196)
    at com.atlassian.confluence.impl.backuprestore.restore.events.OnRestoreEventsSender.sendFinishSiteImportEvents(OnRestoreEventsSender.java:151)
    at com.atlassian.confluence.impl.backuprestore.restore.events.OnRestoreEventsSender.sendFinishEvents(OnRestoreEventsSender.java:64)
    at com.atlassian.confluence.impl.backuprestore.restore.RestoreService.doRestore(RestoreService.java:232)
    at com.atlassian.confluence.impl.backuprestore.restore.SiteRestoreService.doRestoreSynchronously(SiteRestoreService.java:34)
    at com.atlassian.confluence.impl.backuprestore.BackupRestoreJobRunner.runJobSynchronously(BackupRestoreJobRunner.java:68)
    at com.atlassian.confluence.impl.backuprestore.ConfluenceBackupRestoreManager.runJobAndUpdateJobProperties(ConfluenceBackupRestoreManager.java:277)
    at com.atlassian.confluence.impl.backuprestore.ConfluenceBackupRestoreManager.processJobsFromTheQueue(ConfluenceBackupRestoreManager.java:259)
    at com.atlassian.confluence.impl.backuprestore.ConfluenceBackupRestoreManager$$Lambda$7100/0x0000000804dce638.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.7/Executors.java:539)
    at java.util.concurrent.FutureTask.run(java.base@17.0.7/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.7/ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.7/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)

      There are also other threads stuck at initializing com.mysema.query.types.Ops and com.mysema.query.types.OperatorImpl:

      "Indexer: 1" #136848 prio=5 os_prio=0 cpu=22.87ms elapsed=78583.52s tid=0x00007f03795bbc90 nid=0x2fc8 in Object.wait()  [0x00007f03087c3000]
   java.lang.Thread.State: RUNNABLE
    at jdk.internal.misc.Unsafe.ensureClassInitialized0(java.base@17.0.7/Native Method)
    - waiting on the Class initialization monitor for com.mysema.query.types.Ops$MathOps
    at jdk.internal.misc.Unsafe.ensureClassInitialized(java.base@17.0.7/Unsafe.java:1155)
    at jdk.internal.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(java.base@17.0.7/UnsafeFieldAccessorFactory.java:42)
    at jdk.internal.reflect.ReflectionFactory.newFieldAccessor(java.base@17.0.7/ReflectionFactory.java:185)
    at java.lang.reflect.Field.acquireFieldAccessor(java.base@17.0.7/Field.java:1132)
    at java.lang.reflect.Field.getFieldAccessor(java.base@17.0.7/Field.java:1113)
    at java.lang.reflect.Field.get(java.base@17.0.7/Field.java:425)
    at com.mysema.query.types.OperatorImpl.<clinit>(OperatorImpl.java:42)
    at com.mysema.query.types.Ops.<clinit>(Ops.java:30)
    at com.mysema.query.types.Templates.<init>(Templates.java:50)
    at com.mysema.query.types.Templates.<init>(Templates.java:38)
    at com.mysema.query.types.Templates.<clinit>(Templates.java:27)
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper$HibernateDialectToSqlTemplateEnum.<clinit>(QueryDslHelper.java:23)
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper.getSqlTemplateBuilderByDialect(QueryDslHelper.java:46)
    at com.atlassian.confluence.plugins.questions.util.QueryDslHelper.getConfiguration(QueryDslHelper.java:50)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeAnswerRepository.getNumberOfAnswersByQuestionId(ContentTypeAnswerRepository.java:300)
    ...

      "Indexer: 2" #136849 prio=5 os_prio=0 cpu=10.92ms elapsed=78583.52s tid=0x00007f037a206920 nid=0x2fc9 in Object.wait()  [0x00007f0308ecb000]
   java.lang.Thread.State: RUNNABLE
    at com.mysema.query.types.Ops$MathOps.<clinit>(Ops.java:340)
    - waiting on the Class initialization monitor for com.mysema.query.types.OperatorImpl
    at com.mysema.query.types.expr.NumberExpression.<clinit>(NumberExpression.java:46)
    at com.mysema.query.types.path.BeanPath.createNumber(BeanPath.java:257)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.ContentTable.<init>(ContentTable.java:32)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.QueryDSLMapper$11.apply(QueryDSLMapper.java:63)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.QueryDSLMapper$11.apply(QueryDSLMapper.java:60)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.QueryDSLMapper.getMapping(QueryDSLMapper.java:49)
    at com.atlassian.confluence.plugins.questions.repository.querydsl.QueryDSLMapper.getMapping(QueryDSLMapper.java:40)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeFeaturedTopicRepository.getSQLBindings(ContentTypeFeaturedTopicRepository.java:114)
    at com.atlassian.confluence.plugins.questions.contenttype.repository.ContentTypeFeaturedTopicRepository.getFeaturedTopicIds(ContentTypeFeaturedTopicRepository.java:107)
    ...

      Workaround

      Stop Confluence, revert back to it's original state, upgrade Confluence Questions plugin and start Site restore again

      Attachments

        Issue Links

          mentioned in

          Page Loading...

          Page Loading...

          Activity

            People

              d5dce7b13926 agawron
              03cb0c04aa4f Irina Tiapchenko
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: