GC often executing Major or Full collections (similar to previous suggestion) because in fact, the application needs more memory.
Issues with the application like resource contention, long running jobs, expensive computation, …
The intention of this entry is to document a simple process to troubleshoot high CPU usage in Java apps once it has been observed that it’s not related to GC.
3. GENERATE THREAD DUMP OF THE JAVA MAIN THREAD FOUND IN 2.
JVM main thread = Linux process, other Java threads = Lightweight process (LWP)
4. FIND SUSPECT LIGHTWEIGHT PROCESSES (LWPs) WITH HIGH CPU USAGE
It can be observed there are two LWPs using 32% and 14% of the CPU where the rest stays around the same percentage.
5. CONVERT LWP IDs FROM DECIMAL TO HEXADECIMAL FOR PIDs WITH HIGH CPU USAGE
Decimal
Hexadecimal
26605
67ED
26602
67EA
6. OPEN THREADDUMP FROM 3. AND LOCATE HEX VALUES 0x67ed and 0x67ea
7. TROUBLESHOOT
From previous stack trace samples, seems high CPU usage is related to log4j async appender, in this case very likely involving PCIPatternLayout.format() method.
In both instances I was involved with such troubleshooting, the solution was to uplift snmp4j to the latest version at the time of the issue was found and to remove custom layout from log4j appender configuration file for this application.
Thanks for reading and as always, feedback is very much appreciated. If you found this post helpful and would like to receive updates when content like this gets published, sign up to the newsletter.
Orlando L Otero is a Sr Software Engineer Consultant focusing on integration, modernization, cloud adoption and migration, microservices, API design and implementation, and agile delivery.