-
Bug
-
Resolution: Cannot Reproduce
-
High
-
None
-
None
-
18
-
Severity 1 - Critical
-
40
Issue Summary
In some customer environments, Jira may fail to start randomly due to JCMA plugin taking very long time to initialise as observed in the thread dumps during the startup
Steps to Reproduce
Unable to replicate on demand, this is random
Expected Results
Jira should consistently start up every time it is restarted
Actual Results
Jira may fail to startup
The below exception is thrown in the xxxxxxx.log file:
2021-08-01 04:06:38,151-0500 localhost-startStop-1 ERROR [c.a.jira.startup.SystemPluginsEnabledHealthCheck] We couldn't start the following plugins:
- Jira Projects Plugin (com.atlassian.jira.jira-projects-plugin)
- Atlassian Navigation Links Plugin (com.atlassian.plugins.atlassian-nav-links-plugin)
- Atlassian Jira - Plugins - Project Config Plugin (com.atlassian.jira.jira-project-config-plugin)
- Applinks - Plugin - Core (com.atlassian.applinks.applinks-plugin)
- Atlassian OAuth Service Provider Plugin (com.atlassian.oauth.serviceprovider)
- Embedded Gadgets Plugin (com.atlassian.gadgets.embedded)
- Gadget Directory Plugin (com.atlassian.gadgets.directory)
- Atlassian Jira - Plugins - Gadgets Plugin (com.atlassian.jira.gadgets)
- Atlassian Jira - Plugins - Global Issue Navigator (com.atlassian.jira.jira-issue-nav-plugin)
- Atlassian JIRA - Plugins - Quick Edit Plugin (com.atlassian.jira.jira-quick-edit-plugin)
- Atlassian REST - Module Types (com.atlassian.plugins.rest.atlassian-rest-module)
- Atlassian Jira - Plugins - REST Plugin (com.atlassian.jira.rest)
- Opensocial Plugin (com.atlassian.gadgets.opensocial)
- Gadget Dashboard Plugin (com.atlassian.gadgets.dashboard)
The following threads are observed to be interlocking each other:
**In text mode (fore search-ability):
JCMA plugin runnable thread during the startup:
"ThreadPoolAsyncTaskExecutor::Thread 10" #85 prio=5 os_prio=0 cpu=2454.24ms elapsed=60.51s tid=0x00007f73e9978000 nid=0xbc8b in Object.wait() [0x00007f73990d4000]
java.lang.Thread.State: RUNNABLE
at com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate.<init>(AbstractRuntimeDelegate.java:76)
at com.sun.ws.rs.ext.RuntimeDelegateImpl.<init>(RuntimeDelegateImpl.java:56)
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@11.0.9/Native Method)
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@11.0.9/NativeConstructorAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@11.0.9/DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(java.base@11.0.9/Constructor.java:490)
at java.lang.Class.newInstance(java.base@11.0.9/Class.java:584)
at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:65)
at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:155)
at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105)
at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91)
- locked <0x0000000735d35c50> (a java.lang.Class for javax.ws.rs.ext.RuntimeDelegate)
at javax.ws.rs.core.UriBuilder.newInstance(UriBuilder.java:69)
at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:80)
at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:99)
at com.atlassian.jira.migration.common.Constants.buildPageUrl(Constants.kt:40)
at com.atlassian.jira.migration.common.Constants.buildPageUrl$default(Constants.kt:36)
at com.atlassian.jira.migration.preflight.AppOutdatedCheck$1.invoke(AppOutdatedCheck.kt:25)
at com.atlassian.jira.migration.preflight.AppOutdatedCheck$1.invoke(AppOutdatedCheck.kt:18)
at com.atlassian.jira.migration.preflight.AppOutdatedCheck.<init>(AppOutdatedCheck.kt:27)
at com.atlassian.jira.migration.preflight.AppOutdatedCheck.<init>(AppOutdatedCheck.kt:25)
interlocking on JCMA & SSO & Atlassian REST API plugins:
ThreadPoolAsyncTaskExecutor::Thread 2" #77 prio=5 os_prio=0 cpu=654.55ms elapsed=62.55s tid=0x00007f73e9932000 nid=0xbc83 waiting for monitor entry [0x00007f73998e3000]
java.lang.Thread.State: BLOCKED (on object monitor)
at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:89)
- waiting to lock <0x0000000735d35c50> (a java.lang.Class for javax.ws.rs.ext.RuntimeDelegate)
at javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44)
at com.atlassian.plugins.authentication.impl.basicauth.filter.DisableBasicAuthResponseWriter.<clinit>(DisableBasicAuthResponseWriter.java:42)
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@11.0.9/Native Method
"ThreadPoolAsyncTaskExecutor::Thread 20" #95 prio=5 os_prio=0 cpu=1648.94ms elapsed=62.44s tid=0x00007f73e96b2000 nid=0xbc95 waiting for monitor entry [0x00007f73984d1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at javax.ws.rs.ext.RuntimeDelegate.setInstance(RuntimeDelegate.java:140)
- waiting to lock <0x0000000735d35c50> (a java.lang.Class for javax.ws.rs.ext.RuntimeDelegate)
at com.sun.jersey.core.osgi.OsgiRegistry.hookUp(OsgiRegistry.java:432)
at com.sun.jersey.core.osgi.OsgiRegistry.getInstance(OsgiRegistry.java:119) - locked <0x00000007350e0c58> (a java.lang.Class for com.sun.jersey.core.osgi.OsgiRegistry)
at com.sun.jersey.core.reflection.ReflectionHelper.getOsgiRegistryInstance(ReflectionHelper.java:842)
at com.sun.jersey.spi.service.ServiceFinder.<clinit>(ServiceFinder.java:200)
at com.atlassian.plugins.rest.module.util.RestUrlBuilderImpl.loadServiceFinderClass(RestUrlBuilderImpl.java:57)
at com.atlassian.plugins.rest.module.util.RestUrlBuilderImpl.<init>(RestUrlBuilderImpl.java:18)
Workaround
If JCMA & SSO add-on plugins are not used - please disable them and restart JIRA. This should allow JIRA to start.
To remove the JCMA plugin:
- Stop the Jira application.
- Navigate to <jira_install>/atlassian-jira/WEB-INF/atlassian-bundled-plugins path.
- Locate JCMA plugin (plugin name will be like jira-migration-plugin-1.4.4.jar - version number of jar file could be different on your Jira).
- Remove this plugin from <jira_install>/atlassian-jira/WEB-INF/atlassian-bundled-plugins path. (You can place the JCMA jar in another "temp" folder outside the Jira Installation dir).
- If you have upgraded the bundled JCMA plugin, you may also need to navigate to the path <jira_home>/plugins/installed-plugins (or <jira_shared_home>/plugins/installed-plugins in a clustered Jira Data Center environment) and repeat steps 3 and 4.
- Clear the plugin cache.
Clear the following plugin cache files:
- Delete the <jira_home>/plugins/.bundled-plugins directory (this will be regenerated on restart)
- Delete the <jira_home>/plugins/.osgi-plugins directory (this will be regenerated on restart)
- Delete the contents of <jira_install>/work directory (just the contents, not the folder itself)
- Delete the contents of <jira_install>/temp directory (just the contents, not the folder itself)
- Start the Jira application.
- is blocked by
-
-
- Closed
-
- was cloned as
-
-
- Closed
-
- mentioned in
-
Page Failed to load
-
Page Failed to load
-
Page Failed to load
-
Page Loading...
-
-
- Wiki Page
-
[MIG-740] JCMA (Jira Cloud Migration Assistant) plugin may prevent JIRA from starting correctly
The JCMA certainly causes this issue in Jira 8.22.6: https://confluence.atlassian.com/jirakb/jira-data-center-database-import-fails-due-to-missing-tables-fromjira-cloud-migration-assistant-plugin-1115134546.html
As we have no interested in Atlassian's Cloud services, we remove the JCMA plugin as a matter of course whenever we re-provision a Jira node from Terraform.
Bug triggered for me when setting up a new instance of Jira 8.22.2. Took 3 days to revolve this issue. It seems as if you've used a fork of your Java code (which may have a nasty bug in it and causes it to fail on certain systems for reasons unknown).
"atlassian-authentication-plugin-4.2.6-fork-1.1.0.jar" CRC32 checksum a3057615.
Update - 25 April 2022
We're closing this ticket as the issue is not reproducible. If you're experiencing similar issue with latest versions of JCMA, please respond back on this ticket.
Thanks.
Radhika Patodiya
Appeared in Jira DC 8.20.10, JCMA 1.10.13, confirmed by support team.