Recommendation

We should include scripts used to capture thread dumps in the product installer we ship. As an admin, having these scripts available at the ready during a performance incident will provide ease of mind and ease of capture, particularly should another staff member be monitoring the instance at any given time.

When the product is unresponsive, the option to capture thread dumps inside of a support zip may not be available through the web interface.

How to Capture Thread Dumps - Linux

• Generating A Thread Dump

Manual commands

Option 1 - jstack

Option 2 - kill -3

Bash Script

#!/bin/bash
#
# Thread dump script
# Captures one thread dump every 10 seconds, 6 times
# Thus, generating 6 thread dumps over the course of a minute
# Replace "jira" in the APP_NAME declaration below for different products
# This must be run as the same OS user the app runs as
# Include the full path to jstack if it reports the command is not found
# ex: JSTACK_PATH=/opt/java/jdk8u192-b12/bin/jstack

APP_NAME=jira
APP_PID=`ps aux | grep java | grep $APP_NAME | grep -v grep | awk '{print $2}'`
JSTACK_PATH=jstack

echo "Please wait 60 seconds for collection to complete"

for i in $(seq 6)
        do
                echo "Recording thread dump #" ${i}
                top -b -H -p $APP_PID -n 1 > jira_cpu_usage.`date +%s`.txt
                $JSTACK_PATH $APP_PID > jira_threads.`date +%s`.txt
                sleep 10
                done

echo "Done"

How to Capture Thread Dumps - Windows

1. Download PsExec from Sysinternals
2. Unzip the PSTools directory to a suitable place on your server.
3. Open the unzipped PSTools folder, hold shift and right-click inside it, then select Open command window here (if you don't have the 'Open command window here' option you can run a command prompt as Administrator and navigate to the location of the PSTools folder using the command prompt)
4. Identify the PID of the java (tomcat) process using Task Manager > Processes
5. At the command prompt type psexec -s jstack XXXX > thread_dump_YY.log (where XXXX is your PID and YY is the number of the thread dumps taken to append to the name of the log file) then hit enter.
   1. If the jstack command is not recognized, you may need to enter the full path such as: psexec -s "C:\Program Files\Java\jdk1.8.0_201\bin\jstack.exe" XXXX > thread_dump_YY.log
6. You'll need repeat the above 5 or 6 times, about 20 or 30 seconds between each execution, changing the YY each time, for example:

   psexec -s jstack 1234 > thread_dump_01.log
   -- wait ~10 seconds
   psexec -s jstack 1234 > thread_dump_02.log
   -- wait ~10 seconds
   psexec -s jstack 1234 > thread_dump_03.log
   -- wait ~10 seconds
   psexec -s jstack 1234 > thread_dump_04.log
   -- wait ~10 seconds
   psexec -s jstack 1234 > thread_dump_05.log
   -- wait ~10 seconds
   psexec -s jstack 1234 > thread_dump_06.log
   

1. You should now have 6 files named thread_dump_YY.log in the PSTools folder.

Note: If you still have issues with jstack, add the full path of your Java 'bin' directory to the server's system path variable. The Java 'bin' directory is usually in Program Files\Java\bin\jdk<version>\bin. If you add this path to the system variable then you will need to restart the command prompt again for the change to be picked up and the jstack command to work.