[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6PF5QTS0CPMP8FPYVSWVS8F":3,"skill-same:01M6PF5QTS0CPMP8FPYVSWVS8F":71},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":21,"linkOnly":22,"markdownContent":23,"markdownBody":24,"frontMatter":25,"refName":26,"commitSha":27,"skillDirectory":28,"forksCount":29,"repositoryHtmlUrl":30,"repositoryHomepageUrl":6,"ownerHtmlUrl":31,"ownerBlogUrl":32,"ownerDescription":33,"updatedAt":34,"install":35,"resources":48},"01M6PF5QTS0CPMP8FPYVSWVS8F","jvm-troubleshoot",null,"openJiuwen-ai\u002Fagent-core-java","openJiuwen-ai","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F264091041?v=4","01M6PF3SW90T9MK2B9PHFDAE9C","openJiuwen","01M6PF3T0P2A1EYWG3JACQS3MQ",49,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002F.claude\u002Fskills\u002Fjvm-troubleshoot\u002FSKILL.md",[],[18,19,20],"troubleshoot","jvm","jvm troubleshoot","2026-11-10 10:12:30",false,"---\nname: jvm-troubleshoot\ndescription: JVM production troubleshooting handbook. Actively applied when an application reports OOM, CPU 100%, frequent GC, thread deadlock, class loading leak, startup failure, response lag, JVM killed inside a container, or module system reflection failure. Keywords: OOM, OutOfMemoryError, CPU high, GC, Full GC, deadlock, jstack, jmap, jstat, JVM tuning, Metaspace, StackOverflow, OOMKilled, container, cgroup, PID 1, JRE-only, InaccessibleObjectException, --add-opens, module system, JPMS. Not applicable to: coding standard issues (use coding-standard), agent team assembly (use agent-team-guide), non-JVM failures, proactive performance tuning and JVM parameter selection (use performance-tuning).\n---\n\n# JVM Production Troubleshooting Handbook\n\nThis skill is organized by \"symptom -> diagnostic workflow -> root cause table -> fix\", covering 7 categories of high-frequency JVM failures. All commands require first running `jps -lvm` to obtain the `\u003Cpid>`.\n\n**Boundary with performance-tuning**:\n- jvm-troubleshoot (this skill): reactive troubleshooting (OOM\u002FCPU 100% has already occurred) -> follow \"symptom -> diagnosis -> fix\"\n- performance-tuning: proactive optimization (selecting GC before deployment, tuning JVM parameters, writing high-performance code) -> follow \"layer -> selection -> optimization -> verification\"\n\n## Failure Category Quick Reference\n\n| Symptom | Jump to |\n|---|---|\n| Application reports `OutOfMemoryError` | [OOM Troubleshooting](#oom-troubleshooting) |\n| CPU usage at 100% | [CPU Troubleshooting](#cpu-100-troubleshooting) |\n| Frequent GC, application lag | [High-Frequency GC Troubleshooting](#high-frequency-gc-troubleshooting) |\n| Slow response, timeout, no error | [Performance Troubleshooting](#performance-troubleshooting) |\n| Thread deadlock, program stuck | [Thread Deadlock Troubleshooting](#thread-deadlock-troubleshooting) |\n| `ClassNotFoundException` \u002F `NoClassDefFoundError` | [Class Loading Troubleshooting](#class-loading-troubleshooting) |\n| `InaccessibleObjectException` \u002F module system reflection failure | [Module System Troubleshooting](#module-system-troubleshooting) |\n| JVM startup failure, immediate exit | [Startup Failure Troubleshooting](#startup-failure-troubleshooting) |\n| JVM killed inside container \u002F `jps` cannot see \u002F JRE-only | [Container Troubleshooting](#container-troubleshooting) |\n\n## OOM Troubleshooting\n\nOOM types and identification:\n\n| OOM Type | Error Keyword | Root Cause | Troubleshooting |\n|---|---|---|---|\n| **Java heap space** | `OutOfMemoryError: Java heap space` | Heap memory insufficient, objects not released | `jmap -dump` to get dump -> MAT to analyze large objects -> check static Map\u002Flist for continuous growth |\n| **GC overhead** | `GC overhead limit exceeded` | GC spending too much time reclaiming too little memory | Same as above; usually a memory leak or heap too small |\n| **Metaspace** | `OutOfMemoryError: Metaspace` | Too much class metadata (dynamically generated classes, ClassLoader leak) | Adjust `-XX:MaxMetaspaceSize`; check dynamic proxies\u002FCGLIB; see [Class Loading Troubleshooting](#class-loading-troubleshooting) |\n| **Direct buffer** | `OutOfMemoryError: Direct buffer memory` | NIO direct memory not released | Check `ByteBuffer.allocateDirect` release; adjust `-XX:MaxDirectMemorySize` |\n| **StackOverflow** | `StackOverflowError` | Recursion too deep or stack frame too large | Check recursion termination condition; adjust `-Xss` to increase stack (symptomatic, not root fix) |\n| **unable to create thread** | `OutOfMemoryError: unable to create new native thread` | Thread count reached OS limit | `jstack` to see thread count; check thread pool leak; adjust `ulimit -u` |\n\n**Stop the bleeding**: First enable `-XX:+HeapDumpOnOutOfMemoryError` to get a dump, then restart. **Root fix**: Use MAT to analyze the dump, find the GC Root chain, and locate the leak point.\n\n## CPU 100% Troubleshooting\n\n**Diagnostic workflow** (4-step identification):\n\n1. `top` to find the **process** PID with high CPU usage\n2. `top -Hp \u003Cpid>` to find the **thread** TID with high CPU usage (decimal)\n3. `printf \"%x\\n\" \u003Ctid>` to convert to hexadecimal\n4. `jstack \u003Cpid> | grep \u003Chexadecimal tid> -A 30` to see what the thread is doing\n\n**Common root causes**:\n\n| Root Cause | Manifestation | Countermeasure |\n|---|---|---|\n| Infinite loop \u002F infinite recursion | jstack stack frames repeating or fixed | Fix code logic, add termination condition |\n| Regex catastrophic backtracking | Stack in `Pattern.matcher` | Simplify regex, or use pre-compiled + timeout |\n| Large nested collection iteration | Stack in `for` \u002F `Stream` | Change algorithm, reduce O(n^2) to O(n) |\n| Serializing large objects | Stack in `ObjectOutputStream` | Limit object size, or use streaming processing |\n| GC threads consuming all CPU | Multiple `GC task` threads with high CPU | Actually a memory issue, see [High-Frequency GC Troubleshooting](#high-frequency-gc-troubleshooting) |\n| Intensive crypto hashing | Stack in `MessageDigest` | Use a faster algorithm, or execute asynchronously |\n\n**Note**: If CPU is high but business logic is not complex, first check whether GC threads are consuming all CPU (check GC logs).\n\n## High-Frequency GC Troubleshooting\n\n**Diagnostic workflow**:\n\n1. Enable GC logs: `-Xlog:gc*:file=gc.log:time,uptime:filecount=10,filesize=10M` (JDK9+)\n2. Analyze logs with GCViewer or gceasy.io\n3. Check key metrics:\n   - **Full GC frequency**: Normal is once every few hours; once every few minutes is problematic\n   - **GC throughput**: GC time ratio should be \u003C 5%; > 10% requires investigation\n   - **Old generation occupancy**: Continuously growing without dropping = memory leak\n4. `jstat -gcutil \u003Cpid> 1000` to observe occupancy changes across generations\n\n**Common root causes and countermeasures**:\n\n| Root Cause | Countermeasure |\n|---|---|\n| Heap too small | Increase `-Xmx` |\n| Object creation too fast | Caching, object pool, `StringBuilder` |\n| Memory leak | Get dump to find leak point (see OOM Troubleshooting) |\n| Large objects promoted directly to old generation | Adjust `-XX:PretenureSizeThreshold` |\n| Metaspace insufficient | Adjust `-XX:MaxMetaspaceSize` (see OOM Metaspace) |\n| `System.gc()` being called | Add `-XX:+DisableExplicitGC` to disable |\n\n**GC collector selection and production parameter configuration**: Proactive configuration belongs to the performance-tuning skill, see `..\u002Fperformance-tuning\u002Freferences\u002Fgc_tuning.md`. This skill only covers reactive troubleshooting.\n\n## Performance Troubleshooting\n\nWhen response is slow but CPU is not high and GC is normal:\n\n1. `jstack \u003Cpid> > thread.txt` to see thread state distribution\n2. Count `BLOCKED` \u002F `WAITING` threads: `grep \"java.lang.Thread.State\" thread.txt | sort | uniq -c`\n3. Many `BLOCKED` -> lock contention, see [Thread Deadlock Troubleshooting](#thread-deadlock-troubleshooting)\n4. Many `WAITING` -> thread pool idle or waiting for external resources (DB\u002FHTTP)\n5. Check if threads are stuck in `SocketRead` -> RPC timeout or slow peer\n6. `jstat -gcutil \u003Cpid> 1000` to rule out GC issues\n\n## Thread Deadlock Troubleshooting\n\n**Diagnosis**: `jstack \u003Cpid> | grep -A 20 \"Found Java deadlock\"` -- jstack automatically detects synchronized deadlocks and prints them.\n\n**synchronized deadlock**:\n- Manifestation: `jstack` reports \"Found 1 deadlock\", two threads waiting for each other's lock\n- Root cause: Nested `synchronized` blocks with inconsistent lock ordering\n- Fix: Unify lock ordering, or use `tryLock(timeout)`\n\n**Lock deadlock (ReentrantLock)**:\n- Manifestation: Thread `WAITING` in `AbstractQueuedSynchronizer`, but jstack **does not report deadlock** (Lock deadlocks are not automatically detected)\n- Diagnosis: Check `jstack` stack for threads stuck in `lock()` calls\n- Fix: Use `tryLock(timeout, unit)` instead of `lock()`, release and retry on timeout\n\n**Thread leak (pseudo-deadlock)**:\n- Manifestation: Thread count keeps growing, eventually `unable to create thread`\n- Diagnosis: `jstack \u003Cpid> | grep \"java.lang.Thread.State\" | wc -l` to count total threads\n- Fix: Check if thread pool is leaking (`new Thread` not in a pool, `submit` without `shutdown`)\n\n## Class Loading Troubleshooting\n\n`ClassNotFoundException` \u002F `NoClassDefFoundError` \u002F `LinkageError`:\n\n1. `jmap -clstats \u003Cpid>` to see ClassLoader count and loaded class count\n2. Abnormally many ClassLoaders -> ClassLoader leak (common in hot deployment, dynamic proxies)\n3. Class not found -> check classpath: `jcmd \u003Cpid> VM.classloader_stats`\n4. Same class with multiple version conflicts -> `jmap -clstats` to see if multiple ClassLoaders load the same class\n\n**Metaspace leak**:\n- Symptom: Metaspace does not drop after Full GC, eventually OOM Metaspace\n- Diagnosis: `jmap -clstats \u003Cpid>` to see if ClassLoader count keeps growing\n- Root cause: Dynamic proxies (CGLIB\u002FByteBuddy) generating new classes each time without caching, or web container hot deployment not cleaning up old ClassLoaders\n- Fix: Cache proxy classes, fix hot deployment leak, adjust `-XX:MaxMetaspaceSize`\n\n## Module System Troubleshooting\n\nFailures specific to JDK 9+ module system (JPMS). Guaranteed to encounter on JDK 17.\n\n### InaccessibleObjectException\n\n**Symptom**: Reflection call reports `java.lang.reflect.InaccessibleObjectException: Unable to make field accessible: module java.base does not \"opens java.lang\" to unnamed module`\n\n**Root cause**: JDK 9+ module system protects non-exported packages; reflection access requires explicit opening.\n\n**Troubleshooting**:\n1. Check which line of the error mentions \"module X does not opens Y\"\n2. Add the corresponding `--add-opens` for that module\n\n**Common `--add-opens` quick reference**:\n\n| Scenario | Error Module | Parameter |\n|---|---|---|\n| Reflective access to `String` internals | `java.base\u002Fjava.lang` | `--add-opens java.base\u002Fjava.lang=ALL-UNNAMED` |\n| Reflective access to collection internals | `java.base\u002Fjava.util` | `--add-opens java.base\u002Fjava.util=ALL-UNNAMED` |\n| Netty NIO access | `java.base\u002Fjava.nio` | `--add-opens java.base\u002Fjava.nio=ALL-UNNAMED` |\n| Reflective access to `Method` | `java.base\u002Fjava.lang.reflect` | `--add-opens java.base\u002Fjava.lang.reflect=ALL-UNNAMED` |\n| Kryo \u002F Gson serialization | `java.base\u002Fjava.lang` etc. | Usually requires multiple `--add-opens` |\n| Spring \u002F Hibernate reflection | Multiple | See framework documentation |\n\n**Configuration location**:\n- Command line: `java --add-opens ... -jar app.jar`\n- MANIFEST.MF: `Add-Opens: java.base\u002Fjava.lang java.base\u002Fjava.util`\n- K8s: `JAVA_OPTS` environment variable\n\n### `--add-opens` vs `--add-exports`\n\n| Parameter | Purpose |\n|---|---|\n| `--add-opens` | Reflective access (open at runtime) |\n| `--add-exports` | Compile-time + runtime access (non-reflective) |\n| `--add-reads` | Let one module read another module |\n\n**Choose**: Use `--add-opens` for reflection scenarios; use `--add-exports` for direct imports.\n\n### Module Conflict\n\n**Symptom**: Startup reports `Module resolution failed` or `module not found`.\n\n**Troubleshooting**:\n- Check if there are multiple versions of the same module\n- Check module-path (`--module-path`) vs classpath\n- Use `java --list-modules` to see loaded modules\n\n### Module System Diagnostic Commands\n\n```bash\n# View module system parameters\njcmd \u003Cpid> VM.system_properties | grep jdk.module\n\n# View loaded modules\njava --list-modules\n\n# Startup debugging\njava --show-module-resolution -jar app.jar\n```\n\n## Container Troubleshooting\n\nJVM failures specific to K8s \u002F Docker containers. For detailed troubleshooting see `references\u002Fcontainer_troubleshooting.md`.\n\n**Quick reference**:\n\n| Symptom | Root Cause | Countermeasure |\n|---|---|---|\n| Pod `OOMKilled` exit 137 | cgroup memory \u003C JVM usage | Use `MaxRAMPercentage` instead of fixed `-Xmx` |\n| `jps` cannot see JVM | Container main process is PID 1 | Use `jcmd 1 ...` |\n| `jcmd` command not found | JRE-only image | Sidecar with JDK \u002F use jattach \u002F use JFR |\n| `Could not reserve enough space` | `-Xmx` exceeds cgroup | Use `MaxRAMPercentage=75.0` |\n| `unable to create new native thread` | Container ulimit tightened | Adjust `ulimit -u` or reduce thread count |\n| JFR \u002F dump lost on disk | Container path not mounted | Mount PVC to `HeapDumpPath` |\n\n## Startup Failure Troubleshooting\n\nJVM exits immediately on startup; check the error keyword:\n\n| Error | Root Cause | Countermeasure |\n|---|---|---|\n| `Could not reserve enough space for object heap` | `-Xmx` exceeds physical memory | Reduce `-Xmx`, or check if other processes are using memory |\n| `Invalid initial heap size` | `-Xms` \u002F `-Xmx` parameter format error | Check parameter units (`4g` not `4096`) |\n| `Incompatible version` | Compile-time JDK version > runtime | Run with same or higher JDK |\n| `Unsupported major.minor version` | Class file version mismatch | Use `javap -verbose X.class` to check major version |\n| `Unable to open jar` | Jar file corrupted or wrong path | Check `-jar` path and jar integrity |\n| `Error: LinkageError` | Class conflict | Check classpath for duplicate classes |\n\n## Command Quick Reference\n\nJVM diagnostic commands organized by scenario:\n\n```bash\n# === Process Overview ===\njps -lvm                                      # List JVM processes\njcmd \u003Cpid> VM.flags                           # View JVM startup parameters\njcmd \u003Cpid> VM.system_properties               # View system properties\n\n# === Heap Analysis ===\njmap -heap \u003Cpid>                              # Heap overview (generation occupancy)\njmap -histo:live \u003Cpid> | head -20             # Large object histogram (top 20)\njmap -dump:format=b,file=heap.hprof \u003Cpid>     # Export heap dump\njmap -finalizer_info \u003Cpid>                    # View finalizer queue\n\n# === Thread Analysis ===\njstack \u003Cpid> > thread.txt                     # Thread stack\njstack -l \u003Cpid>                               # Thread stack with lock info\ntop -Hp \u003Cpid>                                 # Thread-level CPU usage\n\n# === GC Analysis ===\njstat -gcutil \u003Cpid> 1000                      # Generation occupancy + GC count (refresh every second)\njstat -gccause \u003Cpid> 1000                     # Last GC cause\njcmd \u003Cpid> GC.heap_info                       # Heap info\n\n# === Class Analysis ===\njmap -clstats \u003Cpid>                           # ClassLoader statistics\njcmd \u003Cpid> VM.classloader_stats               # ClassLoader status\njmap -permstat \u003Cpid>                          # Permanent generation statistics (JDK7)\n\n# === System Level ===\ntop                                           # Process CPU\u002Fmemory\niostat -x 1                                   # Disk IO\nnetstat -anp | grep \u003Cpid>                     # Network connections\n```\n\n## JVM Parameters\n\nMonitoring parameters commonly enabled during troubleshooting:\n```\n-XX:+HeapDumpOnOutOfMemoryError               # Auto dump on OOM (essential for OOM troubleshooting)\n-XX:HeapDumpPath=\u002Fvar\u002Flog\u002Fdumps\u002F              # Dump path\n-XX:ErrorFile=\u002Fvar\u002Flog\u002Fhs_err_%p.log          # Fatal error log\n-Xlog:gc*:file=gc.log:time,uptime             # GC log\n-XX:+PrintFlagsFinal                          # Print all JVM parameter final values\n```\n\n**Proactive tuning parameters (heap\u002FGC collector\u002FJIT configuration)**: Belongs to the performance-tuning skill, see `..\u002Fperformance-tuning\u002Freferences\u002Fjvm_params.md`.\n\n## Usage\n\n1. **Locate by symptom**: First check the \"Failure Category Quick Reference\" table, jump to the corresponding section.\n2. **Execute diagnostic workflow**: Each section has numbered steps; execute in order.\n3. **Check root cause table**: After diagnosis, find countermeasures in the \"Common Root Causes\" table.\n4. **Command details**: When command output is unclear, Read `references\u002Fdiagnostic_commands.md`.\n5. **GC troubleshooting**: When investigating GC issues, Read `references\u002Fgc_tuning_guide.md` (reactive diagnostic perspective). For proactive selection and configuration, see performance-tuning.\n6. **Container troubleshooting**: For K8s\u002FDocker container failures, Read `references\u002Fcontainer_troubleshooting.md`.\n7. **JFR analysis**: When using JFR for troubleshooting, Read `references\u002Fjfr_analysis.md`.\n8. **Troubleshooting cases**: When an end-to-end reference is needed, Read `references\u002Ftroubleshooting_cases.md`.\n9. **Do not fabricate when uncertain**: JVM behavior is governed by source code and official documentation; this skill does not replace formal documentation.\n\n## Reference Entries\n\n- **Command details**: `references\u002Fdiagnostic_commands.md` (output field meanings and examples for each JVM command)\n- **GC troubleshooting guide**: `references\u002Fgc_tuning_guide.md` (GC problem troubleshooting from a reactive diagnostic perspective)\n- **Container troubleshooting**: `references\u002Fcontainer_troubleshooting.md` (JVM failures inside K8s\u002FDocker containers: OOM killed \u002F PID 1 \u002F JRE-only)\n- **JFR post-incident analysis**: `references\u002Fjfr_analysis.md` (JFR event quick reference, online streaming, JMC flame graphs)\n- **Troubleshooting cases**: `references\u002Ftroubleshooting_cases.md` (4 end-to-end cases: heap leak \u002F CPU 100% \u002F deadlock \u002F Metaspace leak)\n- **Proactive performance tuning**: `..\u002Fperformance-tuning\u002FSKILL.md` (JVM parameter selection, GC collector selection, JIT tuning, JMH benchmarking)\n- Official documentation: `https:\u002F\u002Fdocs.oracle.com\u002Fen\u002Fjava\u002Fjavase\u002F17\u002Fdocs\u002Fspecs\u002Fman\u002F` (JDK 17 tools)\n- GC log analysis: `https:\u002F\u002Fgceasy.io\u002F`\n- Heap dump analysis: MAT (`https:\u002F\u002Feclipse.org\u002Fmat\u002F`)\n- Project coding standards: `..\u002Fcoding-standard\u002FSKILL.md`\n","# JVM Production Troubleshooting Handbook\n\nThis skill is organized by \"symptom -> diagnostic workflow -> root cause table -> fix\", covering 7 categories of high-frequency JVM failures. All commands require first running `jps -lvm` to obtain the `\u003Cpid>`.\n\n**Boundary with performance-tuning**:\n- jvm-troubleshoot (this skill): reactive troubleshooting (OOM\u002FCPU 100% has already occurred) -> follow \"symptom -> diagnosis -> fix\"\n- performance-tuning: proactive optimization (selecting GC before deployment, tuning JVM parameters, writing high-performance code) -> follow \"layer -> selection -> optimization -> verification\"\n\n## Failure Category Quick Reference\n\n| Symptom | Jump to |\n|---|---|\n| Application reports `OutOfMemoryError` | [OOM Troubleshooting](#oom-troubleshooting) |\n| CPU usage at 100% | [CPU Troubleshooting](#cpu-100-troubleshooting) |\n| Frequent GC, application lag | [High-Frequency GC Troubleshooting](#high-frequency-gc-troubleshooting) |\n| Slow response, timeout, no error | [Performance Troubleshooting](#performance-troubleshooting) |\n| Thread deadlock, program stuck | [Thread Deadlock Troubleshooting](#thread-deadlock-troubleshooting) |\n| `ClassNotFoundException` \u002F `NoClassDefFoundError` | [Class Loading Troubleshooting](#class-loading-troubleshooting) |\n| `InaccessibleObjectException` \u002F module system reflection failure | [Module System Troubleshooting](#module-system-troubleshooting) |\n| JVM startup failure, immediate exit | [Startup Failure Troubleshooting](#startup-failure-troubleshooting) |\n| JVM killed inside container \u002F `jps` cannot see \u002F JRE-only | [Container Troubleshooting](#container-troubleshooting) |\n\n## OOM Troubleshooting\n\nOOM types and identification:\n\n| OOM Type | Error Keyword | Root Cause | Troubleshooting |\n|---|---|---|---|\n| **Java heap space** | `OutOfMemoryError: Java heap space` | Heap memory insufficient, objects not released | `jmap -dump` to get dump -> MAT to analyze large objects -> check static Map\u002Flist for continuous growth |\n| **GC overhead** | `GC overhead limit exceeded` | GC spending too much time reclaiming too little memory | Same as above; usually a memory leak or heap too small |\n| **Metaspace** | `OutOfMemoryError: Metaspace` | Too much class metadata (dynamically generated classes, ClassLoader leak) | Adjust `-XX:MaxMetaspaceSize`; check dynamic proxies\u002FCGLIB; see [Class Loading Troubleshooting](#class-loading-troubleshooting) |\n| **Direct buffer** | `OutOfMemoryError: Direct buffer memory` | NIO direct memory not released | Check `ByteBuffer.allocateDirect` release; adjust `-XX:MaxDirectMemorySize` |\n| **StackOverflow** | `StackOverflowError` | Recursion too deep or stack frame too large | Check recursion termination condition; adjust `-Xss` to increase stack (symptomatic, not root fix) |\n| **unable to create thread** | `OutOfMemoryError: unable to create new native thread` | Thread count reached OS limit | `jstack` to see thread count; check thread pool leak; adjust `ulimit -u` |\n\n**Stop the bleeding**: First enable `-XX:+HeapDumpOnOutOfMemoryError` to get a dump, then restart. **Root fix**: Use MAT to analyze the dump, find the GC Root chain, and locate the leak point.\n\n## CPU 100% Troubleshooting\n\n**Diagnostic workflow** (4-step identification):\n\n1. `top` to find the **process** PID with high CPU usage\n2. `top -Hp \u003Cpid>` to find the **thread** TID with high CPU usage (decimal)\n3. `printf \"%x\\n\" \u003Ctid>` to convert to hexadecimal\n4. `jstack \u003Cpid> | grep \u003Chexadecimal tid> -A 30` to see what the thread is doing\n\n**Common root causes**:\n\n| Root Cause | Manifestation | Countermeasure |\n|---|---|---|\n| Infinite loop \u002F infinite recursion | jstack stack frames repeating or fixed | Fix code logic, add termination condition |\n| Regex catastrophic backtracking | Stack in `Pattern.matcher` | Simplify regex, or use pre-compiled + timeout |\n| Large nested collection iteration | Stack in `for` \u002F `Stream` | Change algorithm, reduce O(n^2) to O(n) |\n| Serializing large objects | Stack in `ObjectOutputStream` | Limit object size, or use streaming processing |\n| GC threads consuming all CPU | Multiple `GC task` threads with high CPU | Actually a memory issue, see [High-Frequency GC Troubleshooting](#high-frequency-gc-troubleshooting) |\n| Intensive crypto hashing | Stack in `MessageDigest` | Use a faster algorithm, or execute asynchronously |\n\n**Note**: If CPU is high but business logic is not complex, first check whether GC threads are consuming all CPU (check GC logs).\n\n## High-Frequency GC Troubleshooting\n\n**Diagnostic workflow**:\n\n1. Enable GC logs: `-Xlog:gc*:file=gc.log:time,uptime:filecount=10,filesize=10M` (JDK9+)\n2. Analyze logs with GCViewer or gceasy.io\n3. Check key metrics:\n   - **Full GC frequency**: Normal is once every few hours; once every few minutes is problematic\n   - **GC throughput**: GC time ratio should be \u003C 5%; > 10% requires investigation\n   - **Old generation occupancy**: Continuously growing without dropping = memory leak\n4. `jstat -gcutil \u003Cpid> 1000` to observe occupancy changes across generations\n\n**Common root causes and countermeasures**:\n\n| Root Cause | Countermeasure |\n|---|---|\n| Heap too small | Increase `-Xmx` |\n| Object creation too fast | Caching, object pool, `StringBuilder` |\n| Memory leak | Get dump to find leak point (see OOM Troubleshooting) |\n| Large objects promoted directly to old generation | Adjust `-XX:PretenureSizeThreshold` |\n| Metaspace insufficient | Adjust `-XX:MaxMetaspaceSize` (see OOM Metaspace) |\n| `System.gc()` being called | Add `-XX:+DisableExplicitGC` to disable |\n\n**GC collector selection and production parameter configuration**: Proactive configuration belongs to the performance-tuning skill, see `..\u002Fperformance-tuning\u002Freferences\u002Fgc_tuning.md`. This skill only covers reactive troubleshooting.\n\n## Performance Troubleshooting\n\nWhen response is slow but CPU is not high and GC is normal:\n\n1. `jstack \u003Cpid> > thread.txt` to see thread state distribution\n2. Count `BLOCKED` \u002F `WAITING` threads: `grep \"java.lang.Thread.State\" thread.txt | sort | uniq -c`\n3. Many `BLOCKED` -> lock contention, see [Thread Deadlock Troubleshooting](#thread-deadlock-troubleshooting)\n4. Many `WAITING` -> thread pool idle or waiting for external resources (DB\u002FHTTP)\n5. Check if threads are stuck in `SocketRead` -> RPC timeout or slow peer\n6. `jstat -gcutil \u003Cpid> 1000` to rule out GC issues\n\n## Thread Deadlock Troubleshooting\n\n**Diagnosis**: `jstack \u003Cpid> | grep -A 20 \"Found Java deadlock\"` -- jstack automatically detects synchronized deadlocks and prints them.\n\n**synchronized deadlock**:\n- Manifestation: `jstack` reports \"Found 1 deadlock\", two threads waiting for each other's lock\n- Root cause: Nested `synchronized` blocks with inconsistent lock ordering\n- Fix: Unify lock ordering, or use `tryLock(timeout)`\n\n**Lock deadlock (ReentrantLock)**:\n- Manifestation: Thread `WAITING` in `AbstractQueuedSynchronizer`, but jstack **does not report deadlock** (Lock deadlocks are not automatically detected)\n- Diagnosis: Check `jstack` stack for threads stuck in `lock()` calls\n- Fix: Use `tryLock(timeout, unit)` instead of `lock()`, release and retry on timeout\n\n**Thread leak (pseudo-deadlock)**:\n- Manifestation: Thread count keeps growing, eventually `unable to create thread`\n- Diagnosis: `jstack \u003Cpid> | grep \"java.lang.Thread.State\" | wc -l` to count total threads\n- Fix: Check if thread pool is leaking (`new Thread` not in a pool, `submit` without `shutdown`)\n\n## Class Loading Troubleshooting\n\n`ClassNotFoundException` \u002F `NoClassDefFoundError` \u002F `LinkageError`:\n\n1. `jmap -clstats \u003Cpid>` to see ClassLoader count and loaded class count\n2. Abnormally many ClassLoaders -> ClassLoader leak (common in hot deployment, dynamic proxies)\n3. Class not found -> check classpath: `jcmd \u003Cpid> VM.classloader_stats`\n4. Same class with multiple version conflicts -> `jmap -clstats` to see if multiple ClassLoaders load the same class\n\n**Metaspace leak**:\n- Symptom: Metaspace does not drop after Full GC, eventually OOM Metaspace\n- Diagnosis: `jmap -clstats \u003Cpid>` to see if ClassLoader count keeps growing\n- Root cause: Dynamic proxies (CGLIB\u002FByteBuddy) generating new classes each time without caching, or web container hot deployment not cleaning up old ClassLoaders\n- Fix: Cache proxy classes, fix hot deployment leak, adjust `-XX:MaxMetaspaceSize`\n\n## Module System Troubleshooting\n\nFailures specific to JDK 9+ module system (JPMS). Guaranteed to encounter on JDK 17.\n\n### InaccessibleObjectException\n\n**Symptom**: Reflection call reports `java.lang.reflect.InaccessibleObjectException: Unable to make field accessible: module java.base does not \"opens java.lang\" to unnamed module`\n\n**Root cause**: JDK 9+ module system protects non-exported packages; reflection access requires explicit opening.\n\n**Troubleshooting**:\n1. Check which line of the error mentions \"module X does not opens Y\"\n2. Add the corresponding `--add-opens` for that module\n\n**Common `--add-opens` quick reference**:\n\n| Scenario | Error Module | Parameter |\n|---|---|---|\n| Reflective access to `String` internals | `java.base\u002Fjava.lang` | `--add-opens java.base\u002Fjava.lang=ALL-UNNAMED` |\n| Reflective access to collection internals | `java.base\u002Fjava.util` | `--add-opens java.base\u002Fjava.util=ALL-UNNAMED` |\n| Netty NIO access | `java.base\u002Fjava.nio` | `--add-opens java.base\u002Fjava.nio=ALL-UNNAMED` |\n| Reflective access to `Method` | `java.base\u002Fjava.lang.reflect` | `--add-opens java.base\u002Fjava.lang.reflect=ALL-UNNAMED` |\n| Kryo \u002F Gson serialization | `java.base\u002Fjava.lang` etc. | Usually requires multiple `--add-opens` |\n| Spring \u002F Hibernate reflection | Multiple | See framework documentation |\n\n**Configuration location**:\n- Command line: `java --add-opens ... -jar app.jar`\n- MANIFEST.MF: `Add-Opens: java.base\u002Fjava.lang java.base\u002Fjava.util`\n- K8s: `JAVA_OPTS` environment variable\n\n### `--add-opens` vs `--add-exports`\n\n| Parameter | Purpose |\n|---|---|\n| `--add-opens` | Reflective access (open at runtime) |\n| `--add-exports` | Compile-time + runtime access (non-reflective) |\n| `--add-reads` | Let one module read another module |\n\n**Choose**: Use `--add-opens` for reflection scenarios; use `--add-exports` for direct imports.\n\n### Module Conflict\n\n**Symptom**: Startup reports `Module resolution failed` or `module not found`.\n\n**Troubleshooting**:\n- Check if there are multiple versions of the same module\n- Check module-path (`--module-path`) vs classpath\n- Use `java --list-modules` to see loaded modules\n\n### Module System Diagnostic Commands\n\n```bash\n# View module system parameters\njcmd \u003Cpid> VM.system_properties | grep jdk.module\n\n# View loaded modules\njava --list-modules\n\n# Startup debugging\njava --show-module-resolution -jar app.jar\n```\n\n## Container Troubleshooting\n\nJVM failures specific to K8s \u002F Docker containers. For detailed troubleshooting see `references\u002Fcontainer_troubleshooting.md`.\n\n**Quick reference**:\n\n| Symptom | Root Cause | Countermeasure |\n|---|---|---|\n| Pod `OOMKilled` exit 137 | cgroup memory \u003C JVM usage | Use `MaxRAMPercentage` instead of fixed `-Xmx` |\n| `jps` cannot see JVM | Container main process is PID 1 | Use `jcmd 1 ...` |\n| `jcmd` command not found | JRE-only image | Sidecar with JDK \u002F use jattach \u002F use JFR |\n| `Could not reserve enough space` | `-Xmx` exceeds cgroup | Use `MaxRAMPercentage=75.0` |\n| `unable to create new native thread` | Container ulimit tightened | Adjust `ulimit -u` or reduce thread count |\n| JFR \u002F dump lost on disk | Container path not mounted | Mount PVC to `HeapDumpPath` |\n\n## Startup Failure Troubleshooting\n\nJVM exits immediately on startup; check the error keyword:\n\n| Error | Root Cause | Countermeasure |\n|---|---|---|\n| `Could not reserve enough space for object heap` | `-Xmx` exceeds physical memory | Reduce `-Xmx`, or check if other processes are using memory |\n| `Invalid initial heap size` | `-Xms` \u002F `-Xmx` parameter format error | Check parameter units (`4g` not `4096`) |\n| `Incompatible version` | Compile-time JDK version > runtime | Run with same or higher JDK |\n| `Unsupported major.minor version` | Class file version mismatch | Use `javap -verbose X.class` to check major version |\n| `Unable to open jar` | Jar file corrupted or wrong path | Check `-jar` path and jar integrity |\n| `Error: LinkageError` | Class conflict | Check classpath for duplicate classes |\n\n## Command Quick Reference\n\nJVM diagnostic commands organized by scenario:\n\n```bash\n# === Process Overview ===\njps -lvm                                      # List JVM processes\njcmd \u003Cpid> VM.flags                           # View JVM startup parameters\njcmd \u003Cpid> VM.system_properties               # View system properties\n\n# === Heap Analysis ===\njmap -heap \u003Cpid>                              # Heap overview (generation occupancy)\njmap -histo:live \u003Cpid> | head -20             # Large object histogram (top 20)\njmap -dump:format=b,file=heap.hprof \u003Cpid>     # Export heap dump\njmap -finalizer_info \u003Cpid>                    # View finalizer queue\n\n# === Thread Analysis ===\njstack \u003Cpid> > thread.txt                     # Thread stack\njstack -l \u003Cpid>                               # Thread stack with lock info\ntop -Hp \u003Cpid>                                 # Thread-level CPU usage\n\n# === GC Analysis ===\njstat -gcutil \u003Cpid> 1000                      # Generation occupancy + GC count (refresh every second)\njstat -gccause \u003Cpid> 1000                     # Last GC cause\njcmd \u003Cpid> GC.heap_info                       # Heap info\n\n# === Class Analysis ===\njmap -clstats \u003Cpid>                           # ClassLoader statistics\njcmd \u003Cpid> VM.classloader_stats               # ClassLoader status\njmap -permstat \u003Cpid>                          # Permanent generation statistics (JDK7)\n\n# === System Level ===\ntop                                           # Process CPU\u002Fmemory\niostat -x 1                                   # Disk IO\nnetstat -anp | grep \u003Cpid>                     # Network connections\n```\n\n## JVM Parameters\n\nMonitoring parameters commonly enabled during troubleshooting:\n```\n-XX:+HeapDumpOnOutOfMemoryError               # Auto dump on OOM (essential for OOM troubleshooting)\n-XX:HeapDumpPath=\u002Fvar\u002Flog\u002Fdumps\u002F              # Dump path\n-XX:ErrorFile=\u002Fvar\u002Flog\u002Fhs_err_%p.log          # Fatal error log\n-Xlog:gc*:file=gc.log:time,uptime             # GC log\n-XX:+PrintFlagsFinal                          # Print all JVM parameter final values\n```\n\n**Proactive tuning parameters (heap\u002FGC collector\u002FJIT configuration)**: Belongs to the performance-tuning skill, see `..\u002Fperformance-tuning\u002Freferences\u002Fjvm_params.md`.\n\n## Usage\n\n1. **Locate by symptom**: First check the \"Failure Category Quick Reference\" table, jump to the corresponding section.\n2. **Execute diagnostic workflow**: Each section has numbered steps; execute in order.\n3. **Check root cause table**: After diagnosis, find countermeasures in the \"Common Root Causes\" table.\n4. **Command details**: When command output is unclear, Read `references\u002Fdiagnostic_commands.md`.\n5. **GC troubleshooting**: When investigating GC issues, Read `references\u002Fgc_tuning_guide.md` (reactive diagnostic perspective). For proactive selection and configuration, see performance-tuning.\n6. **Container troubleshooting**: For K8s\u002FDocker container failures, Read `references\u002Fcontainer_troubleshooting.md`.\n7. **JFR analysis**: When using JFR for troubleshooting, Read `references\u002Fjfr_analysis.md`.\n8. **Troubleshooting cases**: When an end-to-end reference is needed, Read `references\u002Ftroubleshooting_cases.md`.\n9. **Do not fabricate when uncertain**: JVM behavior is governed by source code and official documentation; this skill does not replace formal documentation.\n\n## Reference Entries\n\n- **Command details**: `references\u002Fdiagnostic_commands.md` (output field meanings and examples for each JVM command)\n- **GC troubleshooting guide**: `references\u002Fgc_tuning_guide.md` (GC problem troubleshooting from a reactive diagnostic perspective)\n- **Container troubleshooting**: `references\u002Fcontainer_troubleshooting.md` (JVM failures inside K8s\u002FDocker containers: OOM killed \u002F PID 1 \u002F JRE-only)\n- **JFR post-incident analysis**: `references\u002Fjfr_analysis.md` (JFR event quick reference, online streaming, JMC flame graphs)\n- **Troubleshooting cases**: `references\u002Ftroubleshooting_cases.md` (4 end-to-end cases: heap leak \u002F CPU 100% \u002F deadlock \u002F Metaspace leak)\n- **Proactive performance tuning**: `..\u002Fperformance-tuning\u002FSKILL.md` (JVM parameter selection, GC collector selection, JIT tuning, JMH benchmarking)\n- Official documentation: `https:\u002F\u002Fdocs.oracle.com\u002Fen\u002Fjava\u002Fjavase\u002F17\u002Fdocs\u002Fspecs\u002Fman\u002F` (JDK 17 tools)\n- GC log analysis: `https:\u002F\u002Fgceasy.io\u002F`\n- Heap dump analysis: MAT (`https:\u002F\u002Feclipse.org\u002Fmat\u002F`)\n- Project coding standards: `..\u002Fcoding-standard\u002FSKILL.md`\n",{},"develop","dfece20d51f9b476b87c7fc08dbba71857e1fe17",".claude\u002Fskills\u002Fjvm-troubleshoot",10,"https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java","https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai","https:\u002F\u002Fwww.openjiuwen.com","openJiuwen is an open-source community dedicated to building a production-ready AI Agent platform.","2026-11-10 17:30:15",{"source":7,"skillName":5,"commands":36,"prompt":46,"zipAvailable":47},[37,40,43],{"tool":38,"command":39},"npx","npx skills add openJiuwen-ai\u002Fagent-core-java --skill jvm-troubleshoot",{"tool":41,"command":42},"bunx","bunx skills add openJiuwen-ai\u002Fagent-core-java --skill jvm-troubleshoot",{"tool":44,"command":45},"pnpm","pnpm dlx skills add openJiuwen-ai\u002Fagent-core-java --skill jvm-troubleshoot","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002F.claude\u002Fskills\u002Fjvm-troubleshoot\u002FSKILL.md",true,[49,55,59,63,67],{"path":50,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":53,"sourceHtmlUrl":54},"references\u002Fcontainer_troubleshooting.md","REFERENCE","参考文档",12131,"https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002F.claude\u002Fskills\u002Fjvm-troubleshoot\u002Freferences\u002Fcontainer_troubleshooting.md",{"path":56,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":57,"sourceHtmlUrl":58},"references\u002Fdiagnostic_commands.md",6220,"https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002F.claude\u002Fskills\u002Fjvm-troubleshoot\u002Freferences\u002Fdiagnostic_commands.md",{"path":60,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":61,"sourceHtmlUrl":62},"references\u002Fgc_tuning_guide.md",8140,"https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002F.claude\u002Fskills\u002Fjvm-troubleshoot\u002Freferences\u002Fgc_tuning_guide.md",{"path":64,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":65,"sourceHtmlUrl":66},"references\u002Fjfr_analysis.md",12356,"https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002F.claude\u002Fskills\u002Fjvm-troubleshoot\u002Freferences\u002Fjfr_analysis.md",{"path":68,"resourceType":51,"resourceTypeDesc":52,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Ftroubleshooting_cases.md",8760,"https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002F.claude\u002Fskills\u002Fjvm-troubleshoot\u002Freferences\u002Ftroubleshooting_cases.md",[72,96,107,118,127,136],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":83,"publishedAt":21},"01M6PFBS67R67YZW9V80AHK266","web_research","Search public information and summarize sources with the search tool.","https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002Fexamples\u002Ftoolgroup\u002Fskills\u002Fweb_research\u002FSKILL.md",[78],{"categoryNo":79,"categoryName":80,"taxonomyType":81,"taxonomyTypeDesc":82},"01JBSEEDCAT000000000000021","资料检索与研究","SCENARIO","使用场景",[84,85,86,87,88,89,90,91,92,93,94,95],"web research","research","web","with","information","sources","false","public","tool","examples","search","summarize",{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":105,"publishedAt":21},"01M6PFBNC6CP2CVQE6QPJBZCT3","skill_a","Modified Skill A","https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002Fexamples\u002Fskill_hot_reload\u002Fskills\u002Fskill_a\u002FSKILL.md",[102],{"categoryNo":103,"categoryName":104,"taxonomyType":81,"taxonomyTypeDesc":82},"01JBSEEDCAT000000000000023","Skill与智能体开发",[106],"skill a",{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":114,"publishedAt":21},"01M6PFBHGDK865W5KFD9XC2XPB","skill_30","Skill 30 for hot-reload demo","https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002Fexamples\u002Fskill_hot_reload\u002Fskills\u002Fskill_30\u002FSKILL.md",[113],{"categoryNo":103,"categoryName":104,"taxonomyType":81,"taxonomyTypeDesc":82},[115,116,117],"skill 30","for","demo",{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":125,"publishedAt":21},"01M6PFBDBR8BK7Z2N8XR0WV2P2","skill_29","Skill 29 for hot-reload demo","https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002Fexamples\u002Fskill_hot_reload\u002Fskills\u002Fskill_29\u002FSKILL.md",[124],{"categoryNo":103,"categoryName":104,"taxonomyType":81,"taxonomyTypeDesc":82},[126,116,117],"skill 29",{"skillNo":128,"name":129,"nameZh":6,"description":130,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":131,"categories":132,"keywords":134,"publishedAt":21},"01M6PFB9H4ND4RVTPBVC32CPZZ","skill_28","Skill 28 for hot-reload demo","https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002Fexamples\u002Fskill_hot_reload\u002Fskills\u002Fskill_28\u002FSKILL.md",[133],{"categoryNo":103,"categoryName":104,"taxonomyType":81,"taxonomyTypeDesc":82},[135,116,117],"skill 28",{"skillNo":137,"name":138,"nameZh":6,"description":139,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":140,"categories":141,"keywords":143,"publishedAt":21},"01M6PFB5FR4ACYQ2NCGJ51NSJT","skill_27","Skill 27 for hot-reload demo","https:\u002F\u002Fgithub.com\u002FopenJiuwen-ai\u002Fagent-core-java\u002Fblob\u002Fdevelop\u002Fexamples\u002Fskill_hot_reload\u002Fskills\u002Fskill_27\u002FSKILL.md",[142],{"categoryNo":103,"categoryName":104,"taxonomyType":81,"taxonomyTypeDesc":82},[144,117,116],"skill 27"]