Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-61383

JIRA Software can become unresponsive on upgrade or install when there are many custom fields due to restoring and locking all fields

    XMLWordPrintable

    Details

    • Introduced in Version:
      6.04
    • Support reference count:
      19
    • Regular Expression:
      ManagedFieldConfigurationRestorerImpl\.restoreAndLockFields
    • Symptom Severity:
      Severity 2 - Major
    • Occurrence Factor:
      10%
    • UIS:
      27

      Description

      Summary

      When JIRA Software is upgraded or installed, it can cause JIRA to become unresponsive if there are a huge number Field Configurations with large number of Custom Fields while it iterates through them all, restoring and locking CustomField which belongs to Agile. Problem is a factor of #Field_Configurations * #Custom_Fields_per_Configuration * #Agile_Fields

      Environment

      • Large instance > 1k Custom fields and > 500 Field Configurations (FieldLayouts)

      Steps to Reproduce

      Install JIRA Software onto an instance with a huge number of custom fields.

      Expected Results

      It installs without any problems.

      Actual Results

      JIRA becomes unresponsive while it goes through and restores and locks fields. If generating thread dumps or profiling the below stack trace will be present:

      "JIRA-Bootstrap" #21 daemon prio=5 tid=0x00007fd6b41b0800 nid=0x652b runnable [0x00007fd6de670000]
         java.lang.Thread.State: RUNNABLE
      	at java.net.SocketInputStream.socketRead0(Native Method)
      ...
      	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1659)
      	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
      ...
      	at org.ofbiz.core.entity.GenericValue.create(GenericValue.java:97)
      	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.createValue(DefaultOfBizDelegator.java:296)
      	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.createValue(WrappingOfBizDelegator.java:172)
      	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeAndReturnEditableFieldLayout(AbstractFieldLayoutManager.java:184)
      	- locked <0x00000006c7209840> (a com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager)
      	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeEditableFieldLayout(AbstractFieldLayoutManager.java:201)
      ...
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.saveFieldLayout(ManagedFieldConfigurationRestorerImpl.java:296)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfigurationDescription(ManagedFieldConfigurationRestorerImpl.java:249)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfiguration(ManagedFieldConfigurationRestorerImpl.java:237)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreField(ManagedFieldConfigurationRestorerImpl.java:141)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockField(ManagedFieldConfigurationRestorerImpl.java:110)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockFields(ManagedFieldConfigurationRestorerImpl.java:78)
      ...
      	at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StatupTask.run(DefaultInstantUpgradeManager.java:113)
      	at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:46)
      ...
      

      Full stack-trace:

      "JIRA-Bootstrap" #21 daemon prio=5 tid=0x00007fd6b41b0800 nid=0x652b runnable [0x00007fd6de670000]
         java.lang.Thread.State: RUNNABLE
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      	at java.net.SocketInputStream.read(SocketInputStream.java:170)
      	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
      	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
      	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
      	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:272)
      	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1659)
      	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
      	- locked <0x000000070c099f30> (a org.postgresql.core.v3.QueryExecutorImpl)
      	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:510)
      	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)
      	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:332)
      	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
      	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
      	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:673)
      	at org.ofbiz.core.entity.GenericDAO.singleInsert(GenericDAO.java:205)
      	at org.ofbiz.core.entity.GenericDAO.insert(GenericDAO.java:178)
      	at org.ofbiz.core.entity.GenericHelperDAO.create(GenericHelperDAO.java:83)
      	at org.ofbiz.core.entity.GenericDelegator.create(GenericDelegator.java:556)
      	at org.ofbiz.core.entity.GenericDelegator.create(GenericDelegator.java:542)
      	at org.ofbiz.core.entity.GenericValue.create(GenericValue.java:97)
      	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.createValue(DefaultOfBizDelegator.java:296)
      	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.createValue(WrappingOfBizDelegator.java:172)
      	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeAndReturnEditableFieldLayout(AbstractFieldLayoutManager.java:184)
      	- locked <0x00000006c7209840> (a com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager)
      	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeEditableFieldLayout(AbstractFieldLayoutManager.java:201)
      	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
      	at com.sun.proxy.$Proxy402.storeEditableFieldLayout(Unknown Source)
      	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:136)
      	at com.sun.proxy.$Proxy402.storeEditableFieldLayout(Unknown Source)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.saveFieldLayout(ManagedFieldConfigurationRestorerImpl.java:296)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfigurationDescription(ManagedFieldConfigurationRestorerImpl.java:249)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfiguration(ManagedFieldConfigurationRestorerImpl.java:237)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreField(ManagedFieldConfigurationRestorerImpl.java:141)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockField(ManagedFieldConfigurationRestorerImpl.java:110)
      	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockFields(ManagedFieldConfigurationRestorerImpl.java:78)
      	at com.atlassian.greenhopper.Launcher.onFullSystemStart(Launcher.java:422)
      	at com.atlassian.greenhopper.Launcher.access$200(Launcher.java:55)
      	at com.atlassian.greenhopper.Launcher$FullSystemInitialiser. (Launcher.java:452)
      	at com.atlassian.greenhopper.Launcher$FullSystemInitialiser. (Launcher.java:441)
      	at com.atlassian.greenhopper.Launcher$1.create(Launcher.java:153)
      	at com.atlassian.greenhopper.Launcher$1.create(Launcher.java:149)
      	at com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:326)
      	at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:146)
      	at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:115)
      	at com.atlassian.greenhopper.Launcher.onSystemStartPhase(Launcher.java:207)
      	at com.atlassian.greenhopper.Launcher.onStart(Launcher.java:186)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$4.consume(DefaultLifecycleManager.java:310)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$4.consume(DefaultLifecycleManager.java:306)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyLifecyleAware(DefaultLifecycleManager.java:344)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyOnStartIfStartedAndEnabled(DefaultLifecycleManager.java:304)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.access$300(DefaultLifecycleManager.java:50)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$3.evaluate(DefaultLifecycleManager.java:261)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$3.evaluate(DefaultLifecycleManager.java:257)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyLifecycleAwares(DefaultLifecycleManager.java:286)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyStartableLifecycleAwares(DefaultLifecycleManager.java:255)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.startIfApplicationSetup(DefaultLifecycleManager.java:241)
      	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.start(DefaultLifecycleManager.java:230)
      	at com.atlassian.sal.jira.lifecycle.JiraLifecycleManager.onJiraStart(JiraLifecycleManager.java:64)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48)
      	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107)
      	at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
      	at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
      	at com.atlassian.plugin.event.impl.DefaultPluginEventManager.broadcast(DefaultPluginEventManager.java:73)
      	at com.atlassian.jira.upgrade.PluginUpgradeLauncher.start(PluginUpgradeLauncher.java:30)
      	at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:51)
      	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:180)
      	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$319/36282813.run(Unknown Source)
      	at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StatupTask.run(DefaultInstantUpgradeManager.java:113)
      	at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:46)
      	at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:35)
      	at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:173)
      	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:161)
      	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$318/306334754.run(Unknown Source)
      	at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
      	at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:158)
      	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:147)
      	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$50/1558464762.run(Unknown Source)
      	at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
      	at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
      	at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:137)
      	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
      	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$3/1886674281.run(Unknown Source)
      	at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
      	at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
      	at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:115)
      	at com.atlassian.jira.startup.LauncherContextListener$$Lambda$1/306855479.run(Unknown Source)
      	at java.lang.Thread.run(Thread.java:745)
         Locked ownable synchronizers:
      	- <0x00000006c82d19a8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      

      Notes

      On a larger instance when left over the weekend it took several days to finish locking and restoring the fields. After they were locked and restored following starts of JIRA Software proceeding without any problems.

      Related

      Problem is caused by two factors:

      • Slow population of FieldLayout cache, see related JRA-29310
        ...
        	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.loadFieldLayoutItems(AbstractFieldLayoutManager.java:347)
        	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.loadFieldLayout(AbstractFieldLayoutManager.java:390)
        ...
        	at com.atlassian.cache.memory.DelegatingCache$DelegatingLoadingCache.get(DelegatingCache.java:264)
        	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.getRelevantFieldLayout(AbstractFieldLayoutManager.java:301)
        	at com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager.getEditableFieldLayouts(DefaultFieldLayoutManager.java:256)
        ...
        	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfiguration(ManagedFieldConfigurationRestorerImpl.java:185)
        	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreField(ManagedFieldConfigurationRestorerImpl.java:134)
        	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockField(ManagedFieldConfigurationRestorerImpl.java:106)
        	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockFields(ManagedFieldConfigurationRestorerImpl.java:74)
        ....
        
      • Updating of FieldLayouts is expensive, as per JRA-62242.

      Workaround

      Update the field layout descriptions prior to running the upgrade.

      As with any direct database changes we recommend to fully test this in a staging environment and take a backup of production prior to running any of the following steps.

      • Stop JIRA
      • Run the following 2 queries:

      The following query will update FieldLayoutItem descriptions to the new naming convention.

      UPDATE fieldlayoutitem
      SET
        description = replace(description, 'JIRA Agile', 'JIRA Software')
      WHERE description LIKE '%JIRA Agile%';
      

      Update descriptions on the custom fields themselves.

      UPDATE customfield
      SET
        description = replace(description, 'JIRA Agile', 'JIRA Software')
      WHERE description LIKE '%JIRA Agile%';
      
      • Proceed with upgrade as normal.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tom@atlassian.com Tom Davies
                Reporter:
                dcurrie@atlassian.com David Currie
              • Votes:
                8 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: