Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-73257

Plugins fail to start in Jira 8.20

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Highest Highest
    • 9.0.0, 8.22.3, 8.20.10
    • 8.21.1, 8.17.0, 8.17.1, 8.18.0, 8.18.1, 8.18.2, 8.19.0, 8.19.1, 8.20.0, 8.20.1, 8.20.2, 8.21.0, 8.20.3, 8.20.4, 8.22.0, 8.20.5, 8.20.6, 8.20.7, 8.22.1, 8.22.2, 8.20.8, 8.20.9, 8.22.3, 8.22.4, 8.22.5, 8.20.12, 8.22.6
    • Environment - Java
    • 8.17
    • 157
    • Severity 2 - Major
    • 825
    • Hide

      Hi All,

      Thank you everyone for your patience while we investigate and dig further into the other edge cases. Below we describe our findings.

      We were mainly investigating the following two scenarios:

      Scenario1: This behaviour was seen/caused by Java 11

      We were able to reproduce the same behaviour on Java 8, and that helped us discover that atlassian-servlet-plugin prior to version 6.0.1 was using an old instance of atlassian-plugin-scanner which exposed the plugin startup and hence Jira startup to the following bug https://jira.atlassian.com/browse/JRASERVER-74741. As of the latest versions of Jira (9.0.0, 8.22.3, 8.20.10) no system Apps are using that version of the scanner. Please note, that 3rd party Apps might still be using that and it could potentially impact loading the 3rd party Apps.

      We've detailed the description of JRASERVER-74741 and how to identify the potentially problematic apps.

      Scenario2: Jira system plugin startup is prone to lag and delay

      If the application startup encounters some delays, it will cause a chain reaction of timeouts for a number of system plugin startups which will eventually lead to Jira failing to start. This delay can be caused by environmental factors, such as the underlying system being a bit slower. The issue becomes more apparent with atlassian-authentication-plugin versions between 4.2.0 and 4.2.6 This library was using an expensive operation that introduced this delay, which then triggered the chain reaction. This issue has since been fixed in versions 4.2.7 and onwards. If you encounter other plugin startup failures, please refer to our documentation for help https://confluence.atlassian.com/jirakb/plugins-fail-to-start-in-jira-data-center-1167709529.html. For additional visibility, we created a separate bug JRASERVER-74742 with the steps on how to resolve that scenario.

      Next steps

      We resolved two issues that may happen with the plugin system, and their impact was magnified by both lags introduced with atlassian-authentication-plugin versions from 4.2.0 to 4.2.6 and atlassian-plugin-scanner using the old scanner. As both of those issues are fixed now, the original issue described in the ticket is also fixed. With the above information, we conclude our investigation regarding this issue. We will go ahead and mark this issue as fixed.

      If you encounter more issues related to plugin startup, feel free to reach out to our support team.

      Show
      Hi All, Thank you everyone for your patience while we investigate and dig further into the other edge cases. Below we describe our findings. We were mainly investigating the following two scenarios: Scenario1: This behaviour was seen/caused by Java 11 We were able to reproduce the same behaviour on Java 8, and that helped us discover that atlassian-servlet-plugin prior to version 6.0.1 was using an old instance of atlassian-plugin-scanner which exposed the plugin startup and hence Jira startup to the following bug https://jira.atlassian.com/browse/JRASERVER-74741 . As of the latest versions of Jira (9.0.0, 8.22.3, 8.20.10) no system Apps are using that version of the scanner. Please note, that 3rd party Apps might still be using that and it could potentially impact loading the 3rd party Apps. We've detailed the description of JRASERVER-74741 and how to identify the potentially problematic apps. Scenario2: Jira system plugin startup is prone to lag and delay If the application startup encounters some delays, it will cause a chain reaction of timeouts for a number of system plugin startups which will eventually lead to Jira failing to start. This delay can be caused by environmental factors, such as the underlying system being a bit slower. The issue becomes more apparent with atlassian-authentication-plugin versions between 4.2.0 and 4.2.6 This library was using an expensive operation that introduced this delay, which then triggered the chain reaction. This issue has since been fixed in versions 4.2.7 and onwards. If you encounter other plugin startup failures, please refer to our documentation for help https://confluence.atlassian.com/jirakb/plugins-fail-to-start-in-jira-data-center-1167709529.html . For additional visibility, we created a separate bug JRASERVER-74742 with the steps on how to resolve that scenario. Next steps We resolved two issues that may happen with the plugin system, and their impact was magnified by both lags introduced with atlassian-authentication-plugin versions from 4.2.0 to 4.2.6 and atlassian-plugin-scanner using the old scanner. As both of those issues are fixed now, the original issue described in the ticket is also fixed. With the above information, we conclude our investigation regarding this issue. We will go ahead and mark this issue as fixed. If you encounter more issues related to plugin startup, feel free to reach out to our support team .

      Issue Summary

      System plugins fail to start in Jira 8.20.2. This occurs with frequent restarts.
      This issue can happen both on Java 11.x and Java 8.x versions.

      Steps to Reproduce

      1. Kickstart a Jira instance with 8.20.2 version with any database.
      2. Set the Jira with the first-time setup wizard
      3. Stop Jira after the setup is done
      4. Start Jira twice or thrice to reproduce the problem

      Expected Results

      Jira to start with all plugins even after multiple restarts

      Actual Results

      Jira 8.20.2 fails to start all the plugins.

      The below exception is thrown in the log file:

      2021-12-20 10:11:05,091+0100 localhost-startStop-1 ERROR      [c.a.jira.startup.SystemPluginsEnabledHealthCheck] We couldn't start the following plugins:
    	- Jira Projects Plugin (com.atlassian.jira.jira-projects-plugin)
    	- Atlassian Navigation Links Plugin (com.atlassian.plugins.atlassian-nav-links-plugin)
    	- Atlassian Jira - Plugins - Project Config Plugin (com.atlassian.jira.jira-project-config-plugin)
    	- Applinks - Plugin - Core (com.atlassian.applinks.applinks-plugin)
    	- Atlassian OAuth Service Provider Plugin (com.atlassian.oauth.serviceprovider)
    	- Embedded Gadgets Plugin (com.atlassian.gadgets.embedded)
    	- Gadget Directory Plugin (com.atlassian.gadgets.directory)
    	- Atlassian Jira - Plugins - Gadgets Plugin (com.atlassian.jira.gadgets)
    	- Atlassian Jira - Plugins - Global Issue Navigator (com.atlassian.jira.jira-issue-nav-plugin)
    	- Atlassian JIRA - Plugins - Quick Edit Plugin (com.atlassian.jira.jira-quick-edit-plugin)
    	- Atlassian REST - Module Types (com.atlassian.plugins.rest.atlassian-rest-module)
    	- Atlassian Jira - Plugins - REST Plugin (com.atlassian.jira.rest)
    	- Opensocial Plugin (com.atlassian.gadgets.opensocial)
    	- Gadget Dashboard Plugin (com.atlassian.gadgets.dashboard)

      Plugin container issues

      2022-01-04 17:38:18,078+0100 Spring DM Context Creation Timer ERROR      [c.a.p.osgi.factory.OsgiPlugin] Plugin 'eu.softwareplant.bigpicture' never resolved service '&c037' with filter '(objectClass=com.atlassian.upm.api.license.PluginLicenseManager)'
2022-01-04 17:38:18,079+0100 Spring DM Context Creation Timer ERROR      [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'eu.softwareplant.bigpicture'
org.springframework.context.ApplicationContextException: Application context initialization for 'com.softwareplant.app.bigpicture.app-bigpicture-jiraserver' has timed out waiting for (objectClass=com.atlassian.upm.api.license.PluginLicenseManager)
	at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:489)
	at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:54)
	at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:109)
	at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
	at java.base/java.util.TimerThread.run(Timer.java:506)
2022-01-04 17:38:18,084+0100 Spring DM Context Creation Timer ERROR      [o.e.g.b.e.internal.support.ExtenderConfiguration] Application context refresh failed (NonValidatingOsgiBundleXmlApplicationContext(bundle=com.softwareplant.app.bigpicture.app-bigpicture-jiraserver, config=osgibundle:/META-INF/spring/*.xml))
org.springframework.context.ApplicationContextException: Application context initialization for 'com.softwareplant.app.bigpicture.app-bigpicture-jiraserver' has timed out waiting for (objectClass=com.atlassian.upm.api.license.PluginLicenseManager)
	at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:489)
	at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:54)
	at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:109)
	at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
	at java.base/java.util.TimerThread.run(Timer.java:506)
2022-01-04 17:38:18,085+0100 Spring DM Context Creation Timer ERROR      [o.e.g.b.e.i.dependencies.startup.DependencyWaiterApplicationContextExecutor] Unable to create application context for [com.softwareplant.app.bigpicture.app-bigpicture-jiraserver], unsatisfied dependencies: Dependency on [(objectClass=com.atlassian.upm.api.license.PluginLicenseManager)] (from bean [&c037])
org.springframework.context.ApplicationContextException: Application context initialization for 'com.softwareplant.app.bigpicture.app-bigpicture-jiraserver' has timed out waiting for (objectClass=com.atlassian.upm.api.license.PluginLicenseManager)
	at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:489)
	at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:54)
	at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:109)
	at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
	at java.base/java.util.TimerThread.run(Timer.java:506)

      2022-01-04 17:38:20,788+0100 localhost-startStop-1 ERROR      [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: system-plugins) , Desc = We haven't been able to start all the required system plugins , Exception = We couldn't start the following plugins:<br/><ul><li>Jira Projects Plugin (com.atlassian.jira.jira-projects-plugin)</li><li>Atlassian Navigation Links Plugin (com.atlassian.plugins.atlassian-nav-links-plugin)</li><li>Atlassian Jira - Plugins - Project Config Plugin (com.atlassian.jira.jira-project-config-plugin)</li><li>Applinks - Plugin - Core (com.atlassian.applinks.applinks-plugin)</li><li>Atlassian OAuth Service Provider Plugin (com.atlassian.oauth.serviceprovider)</li><li>Embedded Gadgets Plugin (com.atlassian.gadgets.embedded)</li><li>Gadget Directory Plugin (com.atlassian.gadgets.directory)</li><li>Atlassian Jira - Plugins - Gadgets Plugin (com.atlassian.jira.gadgets)</li><li>Atlassian Jira - Plugins - Global Issue Navigator (com.atlassian.jira.jira-issue-nav-plugin)</li><li>Atlassian JIRA - Plugins - Quick Edit Plugin (com.atlassian.jira.jira-quick-edit-plugin)</li><li>Atlassian REST - Module Types (com.atlassian.plugins.rest.atlassian-rest-module)</li><li>Atlassian Jira - Plugins - REST Plugin (com.atlassian.jira.rest)</li><li>Opensocial Plugin (com.atlassian.gadgets.opensocial)</li><li>Gadget Dashboard Plugin (com.atlassian.gadgets.dashboard)</li></ul>)]

      Workaround 1

      Update atlassian-servlet-plugin to v. 6.0.1 (attached) [Administration -> Manage apps -> Manage apps -> Upload app -> Choose file (From my computer) -> atlassian-servlet-plugin-6.0.1.jar -> Upload]

      Workaround 2

      Check if you have any javaagent and bootdelegation parameters. If so, remove them:

      1. Locate where -Datlassian.org.osgi.framework.bootdelegation or -javaagent is being defined. This is typically in the file JIRA_INSTALL/bin/setenv.sh
      2. Remove any -javaagent and -Datlassian.org.osgi.framework.bootdelegation __ parameters
      3. Restart Jira

      To continue using your monitoring agent with Jira, contact the agent vendor to set things up using atlassian.org.osgi.framework.bootdelegation.extra instead.
      For more information, please see the article Adding classes to OSGi boot class loader (BootDelegation)

        1. atlassian-authentication-plugin-4.2.12.jar
          5.17 MB
        2. atlassian-servlet-plugin-6.0.1.jar
          47 kB

            growinski@atlassian.com Greg Rowinski (Inactive)
            f956e0e022e9 skavatekar
            Votes:
            53 Vote for this issue
            Watchers:
            106 Start watching this issue

              Created:
              Updated:
              Resolved: