Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-62486

Users see 500 errors for every request in Jira after a restart due to bad URL requests during startup

    XMLWordPrintable

Details

    • 7.02
    • 121
    • Severity 1 - Critical
    • 5,168
    • Hide
      Atlassian Update – 12 April 2018

      Hello everyone,

      We managed to find root cause of the bug and patched the part where accessing url would cause JIRA to stall.

      The main problem appeared with Johnson filter which is responsible to drop request that shouldn't be handled during system launch or setup. However, it didn't filter requests which couldn't exist (which were malformed etc.).

      There are however still some pages in jira, that accessed before launch, can throw some stacktraces into the logs. However they won't (from now on) threaten any stability. The fix for stacktraces in logs that do not affect instance health would be tracked in a separate ticket.

      Sorry for inconvenience,
      Piotr Suwała,
      Jira BugFix.

      Show
      Atlassian Update – 12 April 2018 Hello everyone, We managed to find root cause of the bug and patched the part where accessing url would cause JIRA to stall. The main problem appeared with Johnson filter which is responsible to drop request that shouldn't be handled during system launch or setup. However, it didn't filter requests which couldn't exist (which were malformed etc.). There are however still some pages in jira, that accessed before launch, can throw some stacktraces into the logs. However they won't (from now on) threaten any stability. The fix for stacktraces in logs that do not affect instance health would be tracked in a separate ticket. Sorry for inconvenience, Piotr Suwała, Jira BugFix.

    Description

      Summary

      Accessing a non-existent url in JIRA during startup will break access to the instance until a restart is performed.

      During the start the following error can be observed when the request is received:

      2017-09-01 14:29:59,923 http-nio-8080-exec-9 ERROR   -   [o.a.c.c.C.[.[localhost].[/].[action]] Servlet.service() for servlet action threw exception
      java.lang.NullPointerException
      	at com.atlassian.jira.config.webwork.DelegatingConfiguration.getImpl(DelegatingConfiguration.java:26)
      	at webwork.config.Configuration.get(Configuration.java:44)
      	at webwork.config.Configuration.getString(Configuration.java:54)
      	at webwork.dispatcher.GenericDispatcher.<clinit>(GenericDispatcher.java:43)
      	at com.atlassian.jira.web.dispatcher.JiraWebworkActionDispatcher.prepareDispatcher(JiraWebworkActionDispatcher.java:196)
      	... 1 filtered
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
      	... 54 filtered
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      	at java.lang.Thread.run(Thread.java:748)
      

      The following error will be thrown in the logging for every new request after the original null pointer above is hit, and the end user will be greeted with a 500 error:

      2017-09-01 14:30:06,534 http-nio-8080-exec-10 ERROR   -   [o.a.c.c.C.[.[localhost].[/].[action]] Servlet.service() for servlet action threw exception
      java.lang.NoClassDefFoundError: Could not initialize class webwork.dispatcher.GenericDispatcher
      	at com.atlassian.jira.web.dispatcher.JiraWebworkActionDispatcher.prepareDispatcher(JiraWebworkActionDispatcher.java:196)
      	at com.atlassian.jira.web.dispatcher.JiraWebworkActionDispatcher.service(JiraWebworkActionDispatcher.java:133)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
      	... 52 filtered
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      	at java.lang.Thread.run(Thread.java:748)
      

      Steps to Reproduce

      1. Start JIRA
      2. Make frequent requests to a url that does not exist on the JIRA instance while JIRA is still initializing (eg. localhost:PORT/something)

      Expected Results

      JIRA should start up normally.

      Actual Results

      JIRA throws an internal server error.

      Notes

      • Seems to be happening only in very specific environmental conditions (eg. Java version, installer type)

      Workaround

      • Block web access to the instance during start up to prevent the bad request from hitting JIRA during startup such as the following examples:
        • Change the port in server.xml during startup or upgrade to avoid users unintentionally accessing URLs which may cause such problems
        • Leverage a firewall on the server hosting JIRA to block the port during startup
        • In cases where a reverse proxy or balancer is in front of JIRA this can be shut down during startup

      NOTE: The same symptoms are sometimes seen due to corrupted plugin caches or /work directory.  If troubleshooting this issue, clearing these directories may resolve the issue:

      • /JIRA_HOME/plugins/.osgi-plugins/
      • /JIRA_HOME/plugins/.bundled-plugins/
      • /JIRA_INSTALL/work/

      Attachments

        1. jira-exception.txt
          54 kB
        2. screenshot-1.png
          screenshot-1.png
          31 kB

        Issue Links

          Activity

            People

              psuwala ΞΔ (Inactive)
              ddiblasio David Di Blasio
              Votes:
              74 Vote for this issue
              Watchers:
              109 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: