Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-61383

JIRA Software can become unresponsive on upgrade or install when there are many custom fields due to restoring and locking all fields

XMLWordPrintable

      Summary

      When JIRA Software is upgraded or installed, it can cause JIRA to become unresponsive if there are a huge number Field Configurations with large number of Custom Fields while it iterates through them all, restoring and locking CustomField which belongs to Agile. Problem is a factor of #Field_Configurations * #Custom_Fields_per_Configuration * #Agile_Fields

      Environment

      • Large instance > 1k Custom fields and > 500 Field Configurations (FieldLayouts)

      Steps to Reproduce

      Install JIRA Software onto an instance with a huge number of custom fields.

      Expected Results

      It installs without any problems.

      Actual Results

      JIRA becomes unresponsive while it goes through and restores and locks fields. If generating thread dumps or profiling the below stack trace will be present:

      "JIRA-Bootstrap" #21 daemon prio=5 tid=0x00007fd6b41b0800 nid=0x652b runnable [0x00007fd6de670000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
...
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1659)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
...
	at org.ofbiz.core.entity.GenericValue.create(GenericValue.java:97)
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.createValue(DefaultOfBizDelegator.java:296)
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.createValue(WrappingOfBizDelegator.java:172)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeAndReturnEditableFieldLayout(AbstractFieldLayoutManager.java:184)
	- locked <0x00000006c7209840> (a com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeEditableFieldLayout(AbstractFieldLayoutManager.java:201)
...
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.saveFieldLayout(ManagedFieldConfigurationRestorerImpl.java:296)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfigurationDescription(ManagedFieldConfigurationRestorerImpl.java:249)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfiguration(ManagedFieldConfigurationRestorerImpl.java:237)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreField(ManagedFieldConfigurationRestorerImpl.java:141)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockField(ManagedFieldConfigurationRestorerImpl.java:110)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockFields(ManagedFieldConfigurationRestorerImpl.java:78)
...
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StatupTask.run(DefaultInstantUpgradeManager.java:113)
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:46)
...

      Full stack-trace:

      "JIRA-Bootstrap" #21 daemon prio=5 tid=0x00007fd6b41b0800 nid=0x652b runnable [0x00007fd6de670000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:272)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1659)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
	- locked <0x000000070c099f30> (a org.postgresql.core.v3.QueryExecutorImpl)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:510)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:332)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:673)
	at org.ofbiz.core.entity.GenericDAO.singleInsert(GenericDAO.java:205)
	at org.ofbiz.core.entity.GenericDAO.insert(GenericDAO.java:178)
	at org.ofbiz.core.entity.GenericHelperDAO.create(GenericHelperDAO.java:83)
	at org.ofbiz.core.entity.GenericDelegator.create(GenericDelegator.java:556)
	at org.ofbiz.core.entity.GenericDelegator.create(GenericDelegator.java:542)
	at org.ofbiz.core.entity.GenericValue.create(GenericValue.java:97)
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.createValue(DefaultOfBizDelegator.java:296)
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.createValue(WrappingOfBizDelegator.java:172)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeAndReturnEditableFieldLayout(AbstractFieldLayoutManager.java:184)
	- locked <0x00000006c7209840> (a com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.storeEditableFieldLayout(AbstractFieldLayoutManager.java:201)
	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
	at com.sun.proxy.$Proxy402.storeEditableFieldLayout(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:136)
	at com.sun.proxy.$Proxy402.storeEditableFieldLayout(Unknown Source)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.saveFieldLayout(ManagedFieldConfigurationRestorerImpl.java:296)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfigurationDescription(ManagedFieldConfigurationRestorerImpl.java:249)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfiguration(ManagedFieldConfigurationRestorerImpl.java:237)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreField(ManagedFieldConfigurationRestorerImpl.java:141)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockField(ManagedFieldConfigurationRestorerImpl.java:110)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockFields(ManagedFieldConfigurationRestorerImpl.java:78)
	at com.atlassian.greenhopper.Launcher.onFullSystemStart(Launcher.java:422)
	at com.atlassian.greenhopper.Launcher.access$200(Launcher.java:55)
	at com.atlassian.greenhopper.Launcher$FullSystemInitialiser. (Launcher.java:452)
	at com.atlassian.greenhopper.Launcher$FullSystemInitialiser. (Launcher.java:441)
	at com.atlassian.greenhopper.Launcher$1.create(Launcher.java:153)
	at com.atlassian.greenhopper.Launcher$1.create(Launcher.java:149)
	at com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:326)
	at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:146)
	at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:115)
	at com.atlassian.greenhopper.Launcher.onSystemStartPhase(Launcher.java:207)
	at com.atlassian.greenhopper.Launcher.onStart(Launcher.java:186)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$4.consume(DefaultLifecycleManager.java:310)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$4.consume(DefaultLifecycleManager.java:306)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyLifecyleAware(DefaultLifecycleManager.java:344)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyOnStartIfStartedAndEnabled(DefaultLifecycleManager.java:304)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.access$300(DefaultLifecycleManager.java:50)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$3.evaluate(DefaultLifecycleManager.java:261)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$3.evaluate(DefaultLifecycleManager.java:257)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyLifecycleAwares(DefaultLifecycleManager.java:286)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyStartableLifecycleAwares(DefaultLifecycleManager.java:255)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.startIfApplicationSetup(DefaultLifecycleManager.java:241)
	at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.start(DefaultLifecycleManager.java:230)
	at com.atlassian.sal.jira.lifecycle.JiraLifecycleManager.onJiraStart(JiraLifecycleManager.java:64)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107)
	at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
	at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
	at com.atlassian.plugin.event.impl.DefaultPluginEventManager.broadcast(DefaultPluginEventManager.java:73)
	at com.atlassian.jira.upgrade.PluginUpgradeLauncher.start(PluginUpgradeLauncher.java:30)
	at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:51)
	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:180)
	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$319/36282813.run(Unknown Source)
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StatupTask.run(DefaultInstantUpgradeManager.java:113)
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:46)
	at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:35)
	at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:173)
	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:161)
	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$318/306334754.run(Unknown Source)
	at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
	at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:158)
	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:147)
	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$50/1558464762.run(Unknown Source)
	at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
	at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
	at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:137)
	at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
	at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$3/1886674281.run(Unknown Source)
	at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
	at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
	at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:115)
	at com.atlassian.jira.startup.LauncherContextListener$$Lambda$1/306855479.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:745)
   Locked ownable synchronizers:
	- <0x00000006c82d19a8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

      Notes

      On a larger instance when left over the weekend it took several days to finish locking and restoring the fields. After they were locked and restored following starts of JIRA Software proceeding without any problems.

      Related

      Problem is caused by two factors:

      • Slow population of FieldLayout cache, see related JRA-29310 
        ...
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.loadFieldLayoutItems(AbstractFieldLayoutManager.java:347)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.loadFieldLayout(AbstractFieldLayoutManager.java:390)
...
	at com.atlassian.cache.memory.DelegatingCache$DelegatingLoadingCache.get(DelegatingCache.java:264)
	at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.getRelevantFieldLayout(AbstractFieldLayoutManager.java:301)
	at com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager.getEditableFieldLayouts(DefaultFieldLayoutManager.java:256)
...
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreFieldConfiguration(ManagedFieldConfigurationRestorerImpl.java:185)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreField(ManagedFieldConfigurationRestorerImpl.java:134)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockField(ManagedFieldConfigurationRestorerImpl.java:106)
	at com.atlassian.greenhopper.manager.issue.fields.ManagedFieldConfigurationRestorerImpl.restoreAndLockFields(ManagedFieldConfigurationRestorerImpl.java:74)
....
      • Updating of FieldLayouts is expensive, as per JRA-62242.

      Workaround

      Update the field layout descriptions prior to running the upgrade.

      As with any direct database changes we recommend to fully test this in a staging environment and take a backup of production prior to running any of the following steps.

      • Stop JIRA
      • Run the following 2 queries:

      The following query will update FieldLayoutItem descriptions to the new naming convention.

      UPDATE fieldlayoutitem
SET
  description = replace(description, 'JIRA Agile', 'JIRA Software')
WHERE description LIKE '%JIRA Agile%';

      Update descriptions on the custom fields themselves.

      UPDATE customfield
SET
  description = replace(description, 'JIRA Agile', 'JIRA Software')
WHERE description LIKE '%JIRA Agile%';
      • Proceed with upgrade as normal.

        1. JIRA 6.4 snapshot.jps
          3.58 MB
          Dave C

              tom@atlassian.com Tom Davies
              dcurrie@atlassian.com Dave C
              Votes:
              8 Vote for this issue
              Watchers:
              22 Start watching this issue

                Created:
                Updated:
                Resolved: