In order to understand what is happening with our application, we'll need to efficiently analyze the generated snapshot. We'll have a lot of information with precise data of all the threads at the time of the dump. However, we'll need to curate the log files, making some filtering and grouping to extract useful hints from the stack trace. Once we prepared the dump we'll be able to analyze the problem using different tools. Let's see how to decipher the content of a sample dump.

3.1. Synchronization Issues

One interesting tip to filter out the stack trace is the state of the thread. We'll mainly focus on RUNNABLE or BLOCKED threads and eventually TIMED_WAITING ones. Those states will point us in the direction of a conflict between two or more threads:

In a deadlock situation in which several threads running hold a synchronized block on a shared object

In thread contention, when a thread is blocked waiting for others to finish. For example, the dump generated in the previous section

3.2. Execution Issues

As a rule of thumb, for abnormally high CPU usage we only need to look at RUNNABLE threads. We'll use thread dumps together with other commands to acquire extra information. One of these commands is top -H -p PID, which displays what threads are consuming the OS resources within that particular process. We also need to look at the internal JVM threads such as GC just in case. On the other hand, when the processing performance is abnormally low, we'll look at BLOCKED threads.

In those cases, a single dump will most surely not be enough to understand what is happening. We'll need a number of dumps at close intervals in order to compare the stacks of the same threads at different times. On the one hand, one snapshot is not always enough to find out the root of the problem. On the other hand, we need to avoid noise between snapshots (too much information).

To understand the threads' evolution over time, a recommended best practice is to take at least 3 dumps, one at every 10 seconds. Another useful tip is to split the dumps into small chunks to avoid crashes loading the files.

3.3. Recommendations

In order to efficiently decipher the root of the problem, we'll need to organize the huge amount of information in the stack trace. Therefore, we'll take into consideration the following recommendations:

In execution issues, capture several snapshots with an interval of 10 seconds will help to focus on the actual problems. It is also recommended to split the files if needed to avoid loading crashes

Use naming when creating new threads to better identify your source code

Depending on the issue, ignore internal JVM processing (for instance GC)

Focus on long-running or blocked threads when issuing abnormal CPU or memory usage

Correlate the thread's stack with CPU processing by using top -H -p PID

And most importantly, use Analyzer tools

Analyzing the Java thread dumps manually could be a tedious activity. For simple applications, it is possible to identify the threads generating the problem. On the other hand, for complex situations, we'll need tools to ease this task. We'll showcase how to use the tools in the next sections, using the dump generated for the sample thread contention.

4. Online Tools

There are several online tools available. When using this kind of software we need to take into account security issues. Remember that we could be sharing the logs with third-party entities.

https://www.baeldung.com/java-analyze-thread-dumps