Ship Tomcat's long running (stuck) thread logging by default

Problem Definition

Tomcat has no automatic way of reporting if there's long running / stuck threads. When troubleshooting performance it can be a lot harder to know what's going on without generating thread dumps. We'd like to automatically get some more information about this.

Suggested Solution

Adding this parameter to the server.xml will log stuck threads in the catalina.out:

<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="120" />

This will log any request thread (eg http thread) that has been processing for more than 120 seconds. This will not log background threads, eg scheduled jobs or any long running operation that processes in the background, like import/export, reindex etc.
If you are expecting to see something in these logs that you do not, that does not mean it is not running for more than 60s. It likely just means it is not running in a request thread. Thread dumps over time are still the best way to diagnose long running threads.

Workaround

Manually add StuckThreadDetectionValve to JIRA_INSTALL/conf/server.xml

• This must be done on each node if using JIRA Data Center

Add the following line to the server.xml file. Add this line BEFORE the </Engine> tag located near the end of the file.

<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="120" />

EXAMPLE:

• Notice the StuckThreadDetectionValve was added before </Engine> and after the "/>" on the prior line

              <Valve className="org.apache.catalina.valves.AccessLogValve"
                     pattern="%a %{jira.request.id}r %{jira.request.username}r %t &quot;%m %U%q %H&quot; %s %b %D &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot; &quot;%{jira.request.assession.id}r&quot;"/>
  <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="120" />
          </Engine>
      </Service>
  </Server>