Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-61383

JIRA Software can become unresponsive on upgrade or install when there are many custom fields due to restoring and locking all fields

    XMLWordPrintable

Details

    Description

      Summary

      When JIRA Software is upgraded or installed, it can cause JIRA to become unresponsive if there are a huge number Field Configurations with large number of Custom Fields while it iterates through them all, restoring and locking CustomField which belongs to Agile. Problem is a factor of #Field_Configurations * #Custom_Fields_per_Configuration * #Agile_Fields

      Environment

      • Large instance > 1k Custom fields and > 500 Field Configurations (FieldLayouts)

      Steps to Reproduce

      Install JIRA Software onto an instance with a huge number of custom fields.

      Expected Results

      It installs without any problems.

      Actual Results

      JIRA becomes unresponsive while it goes through and restores and locks fields. If generating thread dumps or profiling the below stack trace will be present:

      "JIRA-Bootstrap" #21 daemon prio=5 tid=0x00007fd6b41b0800 nid=0x652b runnable [0x00007fd6de670000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
...
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1659)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
...
	at org.ofbiz.core.entity.GenericValue.create(GenericValue.java:97)
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.createValue(DefaultOfBizDelegator.java:296)
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.createValue(WrappingOfBizDelegator.java:172)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeAndReturnEditableFieldLayout(AbstractFieldLayoutManager.java:184)
	- locked <0x00000006c7209840> (a com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeEditableFieldLayout(AbstractFieldLayoutManager.java:201)
...
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.saveFieldLayout(ManagedFieldConfigurationRestorerImpl.java:296)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfigurationDescription(ManagedFieldConfigurationRestorerImpl.java:249)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfiguration(ManagedFieldConfigurationRestorerImpl.java:237)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreField(ManagedFieldConfigurationRestorerImpl.java:141)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockField(ManagedFieldConfigurationRestorerImpl.java:110)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockFields(ManagedFieldConfigurationRestorerImpl.java:78)
...
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StatupTask.run(DefaultInstantUpgradeManager.java:113)
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:46)
...

      Full stack-trace:

      "JIRA-Bootstrap" #21 daemon prio=5 tid=0x00007fd6b41b0800 nid=0x652b runnable [0x00007fd6de670000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:272)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1659)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
	- locked <0x000000070c099f30> (a org.postgresql.core.v3.QueryExecutorImpl)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:510)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:332)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:673)
	at org.ofbiz.core.entity.GenericDAO.singleInsert(GenericDAO.java:205)
	at org.ofbiz.core.entity.GenericDAO.insert(GenericDAO.java:178)
	at org.ofbiz.core.entity.GenericHelperDAO.create(GenericHelperDAO.java:83)
	at org.ofbiz.core.entity.GenericDelegator.create(GenericDelegator.java:556)
	at org.ofbiz.core.entity.GenericDelegator.create(GenericDelegator.java:542)
	at org.ofbiz.core.entity.GenericValue.create(GenericValue.java:97)
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.createValue(DefaultOfBizDelegator.java:296)
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.createValue(WrappingOfBizDelegator.java:172)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeAndReturnEditableFieldLayout(AbstractFieldLayoutManager.java:184)
	- locked <0x00000006c7209840> (a com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeEditableFieldLayout(AbstractFieldLayoutManager.java:201)
	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
	at com.sun.proxy.$Proxy402.storeEditableFieldLayout(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:136)
	at com.sun.proxy.$Proxy402.storeEditableFieldLayout(Unknown Source)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.saveFieldLayout(ManagedFieldConfigurationRestorerImpl.java:296)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfigurationDescription(ManagedFieldConfigurationRestorerImpl.java:249)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfiguration(ManagedFieldConfigurationRestorerImpl.java:237)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreField(ManagedFieldConfigurationRestorerImpl.java:141)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockField(ManagedFieldConfigurationRestorerImpl.java:110)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockFields(ManagedFieldConfigurationRestorerImpl.java:78)
	at com.atlassian.greenhopper.Launcher.onFullSystemStart(Launcher.java:422)
	at com.atlassian.greenhopper.Launcher.access$200(Launcher.java:55)
	at com.atlassian.greenhopper.Launcher$FullSystemInitialiser. (Launcher.java:452)
	at com.atlassian.greenhopper.Launcher$FullSystemInitialiser. (Launcher.java:441)
	at com.atlassian.greenhopper.Launcher$1.create(Launcher.java:153)
	at com.atlassian.greenhopper.Launcher$1.create(Launcher.java:149)
	at com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:326)
	at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:146)
	at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:115)
	at com.atlassian.greenhopper.Launcher.onSystemStartPhase(Launcher.java:207)
	at com.atlassian.greenhopper.Launcher.onStart(Launcher.java:186)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$4.consume(DefaultLifecycleManager.java:310)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$4.consume(DefaultLifecycleManager.java:306)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyLifecyleAware(DefaultLifecycleManager.java:344)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyOnStartIfStartedAndEnabled(DefaultLifecycleManager.java:304)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.access$300(DefaultLifecycleManager.java:50)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$3.evaluate(DefaultLifecycleManager.java:261)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$3.evaluate(DefaultLifecycleManager.java:257)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyLifecycleAwares(DefaultLifecycleManager.java:286)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyStartableLifecycleAwares(DefaultLifecycleManager.java:255)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.startIfApplicationSetup(DefaultLifecycleManager.java:241)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.start(DefaultLifecycleManager.java:230)
	at com.atlassian.sal.jira.lifecycle.JiraLifecycleManager.onJiraStart(JiraLifecycleManager.java:64)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107)
	at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
	at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
	at com.atlassian.plugin.event.impl.DefaultPluginEventManager.broadcast(DefaultPluginEventManager.java:73)
	at com.atlassian.jira.upgrade.PluginUpgradeLauncher.start(PluginUpgradeLauncher.java:30)
	at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:51)
	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:180)
	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$319/36282813.run(Unknown Source)
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StatupTask.run(DefaultInstantUpgradeManager.java:113)
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:46)
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:35)
	at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:173)
	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:161)
	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$318/306334754.run(Unknown Source)
	at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
	at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:158)
	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:147)
	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$50/1558464762.run(Unknown Source)
	at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
	at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
	at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:137)
	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$3/1886674281.run(Unknown Source)
	at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
	at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
	at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:115)
	at com.atlassian.jira.startup.LauncherContextListener$$Lambda$1/306855479.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:745)
   Locked ownable synchronizers:
	- <0x00000006c82d19a8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

      Notes

      On a larger instance when left over the weekend it took several days to finish locking and restoring the fields. After they were locked and restored following starts of JIRA Software proceeding without any problems.

      Related

      Problem is caused by two factors:

      • Slow population of FieldLayout cache, see related JRA-29310 
        ...
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.loadFieldLayoutItems(AbstractFieldLayoutManager.java:347)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.loadFieldLayout(AbstractFieldLayoutManager.java:390)
...
	at com.atlassian.cache.memory.DelegatingCache$DelegatingLoadingCache.get(DelegatingCache.java:264)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.getRelevantFieldLayout(AbstractFieldLayoutManager.java:301)
	at com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager.getEditableFieldLayouts(DefaultFieldLayoutManager.java:256)
...
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfiguration(ManagedFieldConfigurationRestorerImpl.java:185)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreField(ManagedFieldConfigurationRestorerImpl.java:134)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockField(ManagedFieldConfigurationRestorerImpl.java:106)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockFields(ManagedFieldConfigurationRestorerImpl.java:74)
....
      • Updating of FieldLayouts is expensive, as per JRA-62242.

      Workaround

      Update the field layout descriptions prior to running the upgrade.

      As with any direct database changes we recommend to fully test this in a staging environment and take a backup of production prior to running any of the following steps.

      • Stop JIRA
      • Run the following 2 queries:

      The following query will update FieldLayoutItem descriptions to the new naming convention.

      UPDATE fieldlayoutitem
SET
  description = replace(description, 'JIRA Agile', 'JIRA Software')
WHERE description LIKE '%JIRA Agile%';

      Update descriptions on the custom fields themselves.

      UPDATE customfield
SET
  description = replace(description, 'JIRA Agile', 'JIRA Software')
WHERE description LIKE '%JIRA Agile%';
      • Proceed with upgrade as normal.

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. JRASERVER-62242 Updating of Field Layouts is an expensive process

          • Low - Low priority issues
          • Gathering Impact

          RUM-758 Loading...

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JRASERVER-29310 Filter gadgets take several minutes to load after a field configuration context change

          • Highest - Our top priority issues
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. JSWSERVER-11040 JIRA Agile install is not well logged (and took 78 minutes!)

          • Low - Low priority issues
          • Gathering Impact
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          was cloned as

          JSB-106 Loading...

          JSEV-814 Loading...

          RUN-635 Loading...

          SIN-357 Loading...

          Activity

            People

              tom@atlassian.com Tom Davies
              dcurrie@atlassian.com Dave C
              Votes:
              8 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: