-
Suggestion
-
Resolution: Unresolved
-
None
-
23
-
Problem Definition
Upgrading UPM triggers all apps/applications to restart because of the licensing API provided by UPM. This can cause Jira to return 404 and other errors while the UPM upgrade is in progress, effectively causing downtime.
Impact
Depending on the plugins in the instance, this can cause 404 and 500 errors on (at least) the following pages:
- Dashboard.jpsa
- JiraHome.jspa
- Viewing any issue
This effectively renders Jira unavailable for the time it takes to update the UPM, which can be 5-10 minutes.
In some situations, dashboards will not render gadgets and the situation does not recover by itself.
- Embedded plugins and custom plugins do not render their gadgets on dashboards
- Log excerpt:
2022-05-20 10:28:41,122+0000 http-nio-8080-exec-7508 url: /, /secure/MyJiraHome.jspa, /secure/Dashboard.jspa; user: admin WARN admin 628x67403986x10 lir6e2 xxx.xxx.xxx.xxx,yyy.yyy.yyy.yyy / [c.a.g.renderer.internal.GadgetSpecFactoryImpl] Error occurred while retrieving gadget spec for rest/gadgets/1.0/g/com.arsenalesystems.dataplane:arsenaleDataplaneGadget/gadgets/dataplane/gadget.xml
2022-05-20 10:23:36,363+0000 http-nio-8080-exec-7450 url: /secure/AjaxIssueAction.jspa; user: admin WARN admin 623x67396865x8 1tcu78o xxx.xxx.xxx.xxx,yyy.yyy.yyy.yyy /secure/AjaxIssueAction.jspa [c.atlassian.ozymandias.SafePluginPointAccess] Unable to run plugin code because of 'org.eclipse.gemini.blueprint.service.importer.ServiceProxyDestroyedException - service proxy has been destroyed'. 2022-05-20 10:23:36,386+0000 http-nio-8080-exec-7521 url: /internal-error ERROR [c.a.j.web.servlet.InternalServerErrorServlet] {errorId=c5949b5d-f2db-4ca5-8afa-dfe68e3966b4, interpretedMsg=, cause=java.lang.IllegalArgumentException: Root page 'com.atlassian.jira.gadgets:dashboard-page' does not exist!, stacktrace=java.lang.IllegalArgumentException: Root page 'com.atlassian.jira.gadgets:dashboard-page' does not exist! at com.atlassian.plugin.webresource.assembler.DefaultWebResourceAssembler$2.requirePage(DefaultWebResourceAssembler.java:211) [atlassian-plugins-webresource-4.3.6.jar:?] at com.atlassian.jira.web.action.Dashboard.doExecute(Dashboard.java:187) [classes/:?] at webwork.action.ActionSupport.execute(ActionSupport.java:165) [webwork-1.4-atlassian-30.jar:?] at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:63) [jira-api-8.20.6.jar:?] at webwork.interceptor.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:39) [webwork-1.4-atlassian-30.jar:?]
- In those situations, the following can help:
- Disabling / enabling custom plugins
- Clearing plugin cache and restarting Jira for issues with embedded plugins' gadgets
- Detecting disabled plugins / modules in the database:
SELECT * FROM pluginstate where pluginenabled = 'false';- The solution here would be to delete those particular modules and restart Jira:
DELETE FROM pluginstate WHERE pluginkey='<pluginkey from query above>';
- The solution here would be to delete those particular modules and restart Jira:
Suggested Solution
- As a Jira Server admin I want to update UPM, but I don't want all other apps/applications to depend on UPM.
- Provide a way to schedule UPM update during the following Jira restart.
Workaround
Unfortunately there is no workaround at this time. Instances that are affected by this should have the UPM updated in a scheduled maintenance window.
- is related to
-
JRASERVER-72909 Universal Plugin Manager flushing jira.properties cache causing contention and performance problem
-
- Closed
-
-
-
- Gathering Impact
-
-
- Gathering Interest
-
- Gathering Interest
-
- Not Being Considered
-
BSP-538 Loading...
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Gathering Impact
-
-
- Gathering Interest
- causes
-
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
-
-
[JRASERVER-69963] Make UPM update not impact all other installed apps
This keeps occurring on my cluster. I am unable to update the Universal Plugin Manager without it cache errors.
I then have to scale down, remove the plugin, restart node, scale back up.
Most users depend on third party apps for SSO and MFA.
This breaks access for all users for up to 30 minutes till all the necessary Apps are restarted.
Work around would be for the update to be in 2 stages. Update and Restart for the app. With a warning, to the user that this will stop all App alls which may interfere with the normal user experience including but not limited to access to Jira and Tools
Happened to us with LTS 8.20.7
Updated UPM and Jiraserver became unusable.
I restarted our 3 nodes and service was restored.
WE NEED A WARNING ON THIS! If an action is going to take out availability or even possibly take out availability we should see "please do this during a low activity time".
Even after DAYS, updating the UPM broke our Jira Server.
We had to truncate the "plugins" table to get it to start up again.
Every time we update ANY Add-On within Jira now the entire instance returns dead links everywhere for about 10 minutes while the Add-On installs. It's incredibly disruptive.
Our most recent ZDU Upgrade was an utter disaster as the UPM caused the plugins to fail, and having to take servers offline completely to restore the plugin cache folders on the shared home goes completely against the purpose of the ZDU upgrade.
In our case UPM upgrade disabled all addons but forgot to enable Jira Software app afterwards. Manual enabling or reinstalling via UI was not possible. I had to shutdown the server & manually delete rows with plugins state = false.
Workaround suggested by Atlassian: Upgrade the UPM by file & not with the UI.
Jira 8.15.0
We recently encountered an issue where one of our disabled addons was failing to re-enable, in an attempt to resolve this issue and encouraged by the prominent message at the top of manage apps page (recommending UPM update), we attempted UPM update (with a single click that presented no other warnings) which disabled all our addons including native Jira functionality rendering Jira unusable for 2 hours for all our users, until we were able to resolve the issue with support from Atlassian. In high insight UPM update should have been tested in lower environments and scheduled for an out of hours maintenance window. But the key point I want to highlight is that UPM update should also not pose the risk of such devastating impact on the application as rendering it unusable for all users and if it has to carry such risk hen the update prompt should come with appropriate warnings. I was referred to this suggestion by Atlassian support, who seemed familiar with this problem and am surprised to see this scenario being described as by design (with a feature request/suggestion) rather than a bug, which is how I'd see it. Can this please be prioritised and converted to a bug if possible? For those unfortunate enough to encounter same issue below is how it was resolved in our instance...