[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M718N9M8RW75CWDH63Z8C2TB":3,"skill-same:01M718N9M8RW75CWDH63Z8C2TB":66},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":48,"ownerHtmlUrl":49,"ownerBlogUrl":48,"ownerDescription":50,"updatedAt":51,"install":52,"resources":65},"01M718N9M8RW75CWDH63Z8C2TB","cuda-kernel-profiling",null,"Profile and tune Llaminar V2 CUDA kernels using graph-safe PerfStats replay timing, Nsight Systems (nsys), and Nsight Compute (ncu), then validate with the GEMM perf-test harness, the benchmark subcommand, and parity tests. Use when asked to find the slowest CUDA kernel, diagnose occupancy \u002F register pressure \u002F memory-bound stalls, A\u002FB two kernel variants, or close a prefill\u002Fdecode throughput gap while keeping PyTorch parity.","Llaminar\u002Fllaminar","Llaminar","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F247404048?v=4","01M718MTZG2TQSA7DEB5DKQDN1","01M718MV06G9CV8S2JMN9XHX8Y",48,"AGPL-3.0","https:\u002F\u002Fgithub.com\u002FLlaminar\u002Fllaminar\u002Fblob\u002Fmaster\u002F.agents\u002Fcuda-tuning\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"cuda kernel profiling","profiling","cuda","kernel","diagnose","register","perf","harness","graph","throughput","parity","benchmark","safe","close","test harness","decode","when","variants","pytorch","memory","2026-11-10 10:12:30",false,"---\nname: cuda-kernel-profiling\ndescription: Profile and tune Llaminar V2 CUDA kernels using graph-safe PerfStats replay timing, Nsight Systems (nsys), and Nsight Compute (ncu), then validate with the GEMM perf-test harness, the benchmark subcommand, and parity tests. Use when asked to find the slowest CUDA kernel, diagnose occupancy \u002F register pressure \u002F memory-bound stalls, A\u002FB two kernel variants, or close a prefill\u002Fdecode throughput gap while keeping PyTorch parity.\n---\n\n# CUDA Kernel Profiling & Tuning (Llaminar V2)\n\n## Purpose\n\nA repeatable, evidence-driven workflow for making Llaminar V2 CUDA kernels faster\nwithout breaking correctness. It chains four tools:\n\n1. **PerfStats GPU event timing** — production-graph replay timing and structured route\u002Fhost counters.\n2. **`nsys`** — timeline \u002F launch-count view to understand kernel ordering and CPU↔GPU overlap.\n3. **`ncu`** — per-kernel hardware-counter deep dive (occupancy, registers, spills, stalls).\n4. **Perf-test harness + benchmark + parity** — isolate, A\u002FB, and validate the change end-to-end.\n\nThe golden rule of this codebase: **isolated kernel speedups do not always translate\nto full-model speedups.** Always confirm a win on the real `benchmark` subcommand, and\nalways re-run parity before trusting it.\n\n## Production hot-path invariants\n\nCUDA kernel tuning must preserve the production execution architecture, not\ntrade architecture for an isolated microbenchmark win:\n\n- **No dynamic allocation or deallocation.** Hot-path kernels and launch\n  bridges consume persistent workspace bindings. `cudaMalloc`,\n  `cudaMallocAsync`, `cudaFree`, container growth, or hidden scratch allocation\n  belongs in explicit initialization\u002Fworkspace infrastructure only.\n- **No host transfer.** Do not insert H2D\u002FD2H copies, host-visible mirrors, or\n  host polling into execution. Deliberate RAM\u002Fdisk KV-cache tier movement is a\n  transfer-service operation at an explicit lifecycle boundary, not kernel\n  plumbing.\n- **No stream or device synchronization.** `cudaStreamSynchronize`,\n  `cudaDeviceSynchronize`, blocking copies, and event synchronization on the\n  host are forbidden in the hot path. They are permitted only in explicitly\n  scoped test\u002Fprofiler result collection or terminal result surfacing.\n- **No order-dependent atomics in batch-invariant paths.** Grouped verifier,\n  deterministic decode, and batch-invariant prefill kernels must give every\n  output a unique deterministic writer. An atomic is acceptable only when its\n  value is provably independent of inter-thread order, bitwise equivalence is\n  covered by the all-format\u002FM-totality sweep, and profiling proves it is\n  economical.\n- **Use explicit producer\u002Fconsumer events.** A producer launches on an explicit\n  non-null stream and publishes completion by recording an event on that exact\n  stream. A consumer on another stream waits for that event before reading.\n  Never publish a write on a guessed\u002Fdefault stream, transition coherence\n  manually after a blocking sync, or replace the dependency with a global\n  barrier.\n- **Remain graph-capturable end to end.** Persistent buffers, launches, event\n  record\u002Fwait edges, and collectives must be capturable. Homogeneous GPU\n  inference requires one complete captured graph; segmented execution is not a\n  tuning fallback.\n\n---\n\n## Step 0: Establish the target and baseline\n\nBefore touching anything, capture a baseline so every change is measured against it.\n\n```bash\n# Release build of the engine (always Ninja, never limit parallelism)\nninja -C build_v2_release llaminar2\n\n# Baseline throughput (1st \"Throughput\" line = prefill, 2nd = decode)\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m \u003Cmodel>.gguf -d cuda:0 2>\u002Fdev\u002Fnull | grep -iE \"Throughput\"\n```\n\nRecord the prefill\u002Fdecode tok\u002Fs and the noise band (run 2-3×; typical noise is a few\ntok\u002Fs on prefill, \u003C0.5 tok\u002Fs on decode). A change inside the noise band is **not** a win.\n\n> **Parity gate (must stay PASS the whole time):**\n> ```bash\n> ctest --test-dir build_v2_integration -R \"\u003CModel>ParityTest_(Prefill|Decode)Parity.*CUDA\" --output-on-failure\n> ```\n\n---\n\n## Step 1: Find the hotspot with graph-safe PerfStats\n\nPerfStats measures the same production graph topology used for ordinary\ninference. Enable structured output and explicit GPU-event timing around graph\nreplay:\n\n```bash\nLLAMINAR_PERF_STATS_JSON=\u002Ftmp\u002Fcuda-profile.json \\\nLLAMINAR_PERF_STATS_SUMMARY=1 \\\nLLAMINAR_PERF_STATS_GPU_STAGE_TIMING=1 \\\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m \u003Cmodel>.gguf -d cuda:0\n```\n\nRank `stage_gpu.graph_replay.total` and graph\u002Fsegment rows alongside\n`forward_graph`, `forward_pass`, `kernel`, `mtp`, and model-specific route\ncounters. A monolithic captured graph intentionally does not pretend to have\nper-stage event attribution inside replay. Use `nsys` to identify kernels\ninside that graph, then target a single launch with `ncu`.\n\n`LLAMINAR_PROFILING=1` is deprecated. It aliases the PerfStats summary and GPU\nreplay-event request for compatibility, emits a warning, and must never disable\ncapture or select eager execution. New commands and automation must use the\nexplicit `LLAMINAR_PERF_STATS_*` variables.\n\nPick the single highest-time stage that is plausibly tunable (skip stages that are\nalready compute-bound at the math limit, e.g. the big dense GEMMs, unless that's the\nexplicit target).\n\n---\n\n## Profiler attachment and privilege rules\n\n- Resolve Nsight Systems with `command -v nsys`; the devcontainer package\n  installs it as `\u002Fusr\u002Flocal\u002Fbin\u002Fnsys`, independently of the CUDA toolkit.\n  Nsight Compute remains `\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu`.\n- Run both tools through `sudo -E` on hosts where\n  `\u002Fproc\u002Fdriver\u002Fnvidia\u002Fparams` reports `RmProfilingAdminOnly: 1`. An\n  unprivileged `nsys` launch can appear successful while collecting only CUDA\n  graph-construction metadata and no executed kernels.\n- Preserve `LLAMINAR_*` variables across privilege escalation with `sudo -E`,\n  or pass the exact variables after `sudo`. Ordinary `sudo` strips them.\n- Pass `--no-mpi-bootstrap` only when profiling or debugging `llaminar2`\n  directly; otherwise Nsight attaches to the `mpirun` wrapper. Never use the\n  flag for production or canonical benchmark measurements.\n- Do not pass `--no-mpi-bootstrap` to standalone test\u002Fperformance binaries;\n  they do not auto-bootstrap MPI.\n- Write `.nsys-rep` and `.ncu-rep` artifacts under `\u002Ftmp` or another explicit\n  result directory, not in the repository.\n\n### Prove profiler attachment before a model run\n\nDo not spend a model load on an unproven profiler setup. First profile a focused\nintegration test that executes one eager launch and one captured replay:\n\n```bash\nsudo -E \"$(command -v nsys)\" profile \\\n  --trace=cuda,nvtx --sample=none --cpuctxsw=none \\\n  --cuda-graph-trace=node --force-overwrite=true \\\n  --output=\u002Ftmp\u002Fllaminar-nsys-cuda-smoke \\\n  .\u002Fbuild_v2_integration\u002Ftests\u002Fv2\u002Fv2_integration_cuda_moe_kernel \\\n  --gtest_filter=Test__CUDAMoEKernel.RouteWithTensorsTiledPrefillCapturesAfterWarmup\n\n\"$(command -v nsys)\" stats --report cuda_gpu_kern_sum,cuda_api_sum \\\n  \u002Ftmp\u002Fllaminar-nsys-cuda-smoke.nsys-rep\n```\n\nThe smoke report must contain a `CUDA GPU Kernel Summary`, including both\nlaunch instances from the focused test. A report containing\n`CUDA_GRAPH_NODE_EVENTS` or `CUDA_GRAPH_EVENTS` but no\n`CUPTI_ACTIVITY_KIND_KERNEL` is **not** an execution trace. Its diagnostics\nnormally say `CUDA profiling might have not been started correctly` or report\nonly graph-node creation. Check privilege first.\n\nCUDA 13 conditional\u002Fdevice-loop graphs expose a second tool boundary on Ampere:\nthe tested Nsight Systems 2025.3 and 2026.1 releases can collect ordinary eager\nand simple captured graphs yet omit all activities from Llaminar's complete\nproduction conditional graph. Confirm this by comparing the smoke test with the\nreal graph report; never disable production graph execution and present the\neager timeline as production evidence. Use targeted Nsight Compute node\nprofiling below for the real graph.\n\nDo not defer `ncu` attachment with `--profile-from-start off` or a late\n`cudaProfilerStart()` range for this graph topology. That technique profiles a\nsimple already-instantiated graph, but the tested Llaminar conditional graph\nexposes no inner kernel nodes when profiling begins after graph construction.\nKeep NCU active from process start and use its graph-node kernel filter to\nselect one production launch.\n\n---\n\n## Step 2: Timeline sanity check with `nsys`\n\nUse `nsys` to confirm the kernel of interest dominates and to see launch counts \u002F sync stalls.\n\n```bash\n# --no-mpi-bootstrap is REQUIRED so the profiler attaches to llaminar2, not the mpirun wrapper\nsudo -E \"$(command -v nsys)\" profile -t cuda --stats=true -o \u002Ftmp\u002Ftrace -f true \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 oneshot --no-mpi-bootstrap -d cuda:0 \\\n  -m \u003Cmodel>.gguf -p \"test\" -n 10\n\n# Per-kernel summary (gives the launch order needed for ncu --launch-skip)\n\"$(command -v nsys)\" stats --report cuda_gpu_kern_sum \u002Ftmp\u002Ftrace.nsys-rep\n```\n\n`nsys` also tells you the kernel's mangled name and how many times it launches — both\nneeded to target `ncu` precisely.\n\n> `--no-mpi-bootstrap` is ONLY for profiler\u002Fdebugger attach (`ncu`\u002F`nsys`\u002F`gdb`\u002F`perf`).\n> It disables NUMA-aware thread pinning, so it gives misleading *performance* numbers —\n> never use it for benchmarks or production.\n\n### Inventory durations inside the complete production graph with NCU\n\nConditional\u002Fdevice-loop graphs on Ampere may be invisible to Nsight Systems\neven when a simple graph smoke is visible. Use this proven one-counter NCU\ntransaction to enumerate the real production graph before selecting a deep\nprofile. The critical details are: attach to `llaminar2` directly with\n`--no-mpi-bootstrap`, keep profiling active from process start, request graph\n**node** profiling, use application replay, and filter out model-load kernels\nso the launch budget reaches captured inference:\n\n```bash\nRUNTIME_KERNELS='regex:(buildGrouped|build_active|count_per|cuda_attention|cuda_derive_attention|cuda_gated|cuda_gdn|cuda_q_gate|cuda_short_conv1d|exclusive_scan|float_to_int|fp32_|grouped_|groupedImma|quantize_activations|router_gate|scatter_tokens|shared_expert|softmax_topk|embedding_lookup|fused_|mtpConcat|requestTerminal|cuda_kv|residual_add|rmsnorm|rope_|flash_attention|nativeVnniTC|route_logits|shiftedMTP|ring_append|ring_gather)'\n\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --target-processes all \\\n  --replay-mode application \\\n  --graph-profiling node \\\n  --kernel-name-base demangled \\\n  --kernel-name \"$RUNTIME_KERNELS\" \\\n  --launch-count 1200 \\\n  --metrics gpu__time_duration.sum \\\n  --clock-control none \\\n  --force-overwrite \\\n  --export \u002Ftmp\u002Fllaminar-production-node-duration \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 benchmark --no-mpi-bootstrap \\\n    -m \u003Cmodel.gguf> -d cuda:0 \\\n    --prompt-file \u003Cfixed-prompt.txt> \\\n    --seed \u003Cseed> --temperature \u003Ctemperature> \\\n    --mtp --mtp-draft-tokens \u003Cdepth> \\\n    --mtp-depth-policy fixed \\\n    --mtp-verify-mode speculative-sampling \\\n    \u003Cremaining-exact-release-arguments>\n\n\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --import \u002Ftmp\u002Fllaminar-production-node-duration.ncu-rep \\\n  --page raw --csv \\\n  > \u002Ftmp\u002Fllaminar-production-node-duration.csv\n\n# Recover the exact profiler command and target provenance from any report.\n\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --import \u002Ftmp\u002Fllaminar-production-node-duration.ncu-rep \\\n  --page session --csv\n```\n\nChoose `--launch-count` above the expected runtime-node count, then verify the\nCSV reached the terminal graph kernels rather than silently truncating at the\nbudget. Sum `gpu__time_duration.sum` by exact demangled kernel name and retain\ncall counts plus grid\u002Fblock geometry; aggregate names without geometry can\nhide one pathological shape behind many cheap launches. Inspect the CSV units\nrow before summing: NCU auto-scales `gpu__time_duration.sum` and may emit `us`\nfor one report and `ms` for another. Normalize every row to one unit rather than\nassuming a fixed scale. This first pass uses one metric and is for attribution\nonly. It does not replace warmed unprofiled Release timing, and it does not\ncertify occupancy or spills.\n\nIf NCU prints `No kernels were profiled`, first run the profiler-attachment\nsmoke above, then inspect the report's `Available Kernels`. Do not switch to\neager execution, segmented graphs, late `cudaProfilerStart()`, or a host-side\nsurrogate to make the profiler easier to use; those are different execution\npaths.\n\n---\n\n## Step 3: Deep-dive a kernel with `ncu`\n\n`ncu` replays each kernel 4-8× through hardware counters, so always **target one kernel**\nand **skip warmups**.\n\n```bash\n# sudo -E preserves env (LLAMINAR_*); ncu needs sudo for counter access\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --kernel-name \"\u003Cmangled_or_substr>\" \\\n  --launch-skip 1 --launch-count 1 \\\n  --section SpeedOfLight \\\n  --section Occupancy \\\n  --section LaunchStats \\\n  --section MemoryWorkloadAnalysis \\\n  --section WarpStateStats \\\n  --section ComputeWorkloadAnalysis \\\n  --target-processes all \\\n  -o \u002Ftmp\u002Fk_ncu -f \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 oneshot --no-mpi-bootstrap -d cuda:0 \\\n  -m \u003Cmodel>.gguf -p \"test\" -n 1\n\n# Read it back\nsudo \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu -i \u002Ftmp\u002Fk_ncu.ncu-rep --page details\n\n# Inspect the sections supported by the installed Nsight version\n\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu --list-sections\n```\n\n### Profile a node inside the real captured graph\n\n`ncu` has a graph-node profiler independent of the Nsight Systems timeline.\nUse it when the Systems version cannot expand Llaminar's conditional graph:\n\n```bash\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --target-processes all --graph-profiling node \\\n  --kernel-name-base demangled \\\n  --kernel-name 'regex:\u003Cunique-production-kernel-substring>' \\\n  --launch-count 1 \\\n  --section LaunchStats --section Occupancy --section SpeedOfLight \\\n  --section MemoryWorkloadAnalysis \\\n  --clock-control none --force-overwrite \\\n  --export \u002Ftmp\u002Fllaminar-real-node \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 benchmark --no-mpi-bootstrap \u003Cexact-release-args>\n\nsudo \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --import \u002Ftmp\u002Fllaminar-real-node.ncu-rep \\\n  --page details --print-details all\n```\n\nMatch one unique kernel family and set `--launch-count 1`. Default kernel replay\nbacks up device-written memory before each metric pass; with a resident 35B\nmodel this can take roughly a minute even for one node. Keep the section set\nminimal, or collect a one-pass custom metric list when only spill\u002Fvectorization\nproof is needed. Never interpret the profiled duration as canonical latency;\nuse the unprofiled Release benchmark for timing.\n\nKernel families and template winners can change after rebuilding generated\ndispatch. If a selector produces `No kernels were profiled`, read NCU's\n`Available Kernels` list and select the exact current production family. A\ndeliberately impossible selector with a one-pass section such as `LaunchStats`\nis a quick inventory probe. Do not carry yesterday's selector into today's\nprofile without checking it.\n\nFor the Qwen 3.6 35B grouped-prefill path, a proven real-graph selector is\n`regex:groupedImmaProjectionKernel`. `MemoryWorkloadAnalysis` must report zero\n`derived__local_spilling_requests`; `LaunchStats`, `Occupancy`, and\n`SpeedOfLight` supply registers, launch geometry, achieved occupancy, and the\ncompute\u002Fmemory throughput balance.\n\n> 🧹 **MANDATORY CLEANUP after every ncu run** (ncu leaves zombie processes that hold\n> the GPU and corrupt the next run):\n> ```bash\n> sudo pkill -9 -f \"llaminar2 oneshot\"; sudo pkill -9 -f \"ncu --kernel\"; sleep 1; nvidia-smi\n> ```\n\n### What to read first\n\n| Metric | Healthy | Red flag → likely cause |\n|--------|---------|-------------------------|\n| `Local Memory Spilling` | 0 | >0 → register pressure spilling to DRAM |\n| `Registers Per Thread` | matches `__launch_bounds__` | unexpected → check launch config \u002F occupancy ceiling |\n| `Compute (SM) Throughput` | >60% (compute-bound) | \u003C30% → latency-bound |\n| `DRAM Throughput` | \u003C30% (compute-bound) | >60% w\u002F low compute → spilling or bad access pattern |\n| `Achieved Occupancy` | near theoretical | low + low compute → not enough warps to hide latency |\n| `Warp Cycles Per Issued Instruction` | \u003C15 | >30 → severe stalls (check WarpStateStats breakdown) |\n\n| Warp stall dominated by | Cause | Fix direction |\n|-------------------------|-------|---------------|\n| `Barrier` | uneven work across `__syncthreads()` | fewer sync points, warp-level sync |\n| `L1\u002FTEX` + spilling | spilled regs reloaded from local mem | reduce live registers \u002F relax `__launch_bounds__` MIN_BLOCKS |\n| `Long Scoreboard` (memory) | global-load latency | improve coalescing \u002F prefetch \u002F more warps |\n\n### Target standalone and A\u002FB launches\n\nRun `nsys` first and count only launches matching the eventual `--kernel-name`\nfilter. `--launch-skip` counts matching launches, not all launches in the\nprocess. Profile variant A and variant B in separate, otherwise-identical\ninvocations with `--launch-count 1`; do not compare replay duration from the\nreports. Use the saved reports only for counters and resource diagnostics, and\nuse the unprofiled harness for canonical latency.\n\nFor standalone performance binaries, invoke `nsys`\u002F`ncu` on the binary\ndirectly. Keep `--target-processes all` when the harness can fork, preserve its\nshape\u002Fformat\u002FM filters, and recalculate launch skip whenever the warmup or\ncandidate order changes.\n\n### Prove NativeVNNI IMMA and vectorized memory execution\n\nDo not infer tensor-core use from a source-level `mma.sync` spelling or a\ncandidate-family name. Profile the actual production-selected candidate and\nrecord the GA102 IMMA and memory-width counters on one isolated extra launch:\n\n```bash\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --kernel-name \"nativeVnniTC\" --launch-skip 1 --launch-count 1 \\\n  --metrics \\\nsm__inst_executed_pipe_tensor_op_imma.sum,\\\nsmsp__average_inst_executed_pipe_tensor_op_imma_per_warp,\\\nsmsp__sass_inst_executed_op_memory_128b.sum,\\\nsmsp__sass_average_data_bytes_per_sector_mem_global_op_ld.ratio \\\n  --target-processes all -o \u002Ftmp\u002Fnative_vnni_instruction_proof -f \\\n  .\u002Fbuild_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm \\\n  --gtest_filter=\"CUDANativeVNNIGemmPerf.Performance_AllFormats_AllShapes\"\n```\n\nThe command MUST use the harness's normal shape\u002Fformat\u002FM filters so Nsight sees\nthe candidate selected by production dispatch. A nonzero IMMA count proves the\ncompiled integer tensor-core path. The 128-bit instruction count and bytes per\nsector diagnose vectorization and coalescing independently. Nsight replay time\nis never canonical latency and must not enter dispatch labels.\n\n---\n\n## Step 4: Isolate & A\u002FB with the GEMM perf-test harness\n\nFor GEMM\u002FMoE-expert kernels, the perf test gives a fast, correctness-gated A\u002FB loop that\nis far quicker than full-model rebuilds. It shares the production decode\u002Fcodebook helpers,\nso it is a valid proxy for those kernels.\n\n```bash\n# Build the perf test (Release)\nninja -C build_v2_release tests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm\n\n# Correctness gate is cosine >= 0.9990 vs cuBLAS. The full-shape sweep is SLOW\n# (single-threaded CPU reference, several minutes) — ALWAYS scope to the shapes you care about.\nLLAMINAR_CUDA_NATIVE_GEMM_SHAPES=\"35BMoE_Expert_GateUp,35BMoE_Expert_Down\" \\\n  .\u002Fbuild_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm --gtest_filter=\"*\"\n```\n\nUseful env filters (comma-separated, case-insensitive): `LLAMINAR_CUDA_NATIVE_GEMM_FORMATS`,\n`LLAMINAR_CUDA_NATIVE_GEMM_SHAPES`, `LLAMINAR_CUDA_NATIVE_GEMM_PREFILL_M`,\n`LLAMINAR_CUDA_NATIVE_GEMM_BENCH`, `LLAMINAR_CUDA_NATIVE_GEMM_WORKERS`.\n\nAdd new production shapes to `kQwenShapes` in\n`tests\u002Fv2\u002Fperformance\u002Fkernels\u002Fcuda\u002Fgemm\u002FCUDANativeVNNIGemmPerfCommon.h` so the harness\ncovers the exact GEMMs a model actually runs.\n\n### Generated GEMM\u002FGEMV dispatch training pipeline\n\nRead `.agents\u002Fnativevnni-gemm-tuning\u002FSKILL.md` for the shared corpus, learner,\ncertification, Git LFS, and installation workflow. This section owns only\nCUDA-specific kernel\u002Fprofiler constraints.\n\nDo not land source-level \"one shape gets this tile\" overrides for CUDA NativeVNNI\nunless the user explicitly asks for a temporary experiment. The durable path is:\n\n1. Add or confirm the production shapes in\n   `tests\u002Fv2\u002Fperformance\u002Fkernels\u002Fcuda\u002Fgemm\u002FCUDANativeVNNIGemmPerfCommon.h`.\n2. For decode\u002FGEMV dispatch tables, use the Git-LFS-aware turnkey transaction:\n   `scripts\u002Ftrain_native_vnni_dispatch.sh --backend cuda --install`.\n   The lower-level `refresh_native_vnni_dispatch_tables.sh` remains the phase\n   executor used by the driver and for focused diagnostic collection.\n   It first sweeps and freezes `Fast M=1`, then certifies grouped verifier\n   `M=2..16,31` against that exact staged serial policy. Fifteen drafts require\n   verifier M16 because the target pass includes one bonus row; M31 is a deeper\n   sentinel, not a runtime maximum. Installation requires this complete matrix.\n   The M1 broad sweep runs complete sample-interleaved candidate rounds. Failed\n   development-CV cells are converted by\n   `tests\u002Fv2\u002Fperformance\u002Fkernels\u002Fnative_vnni_dispatch\u002Fpaired_requests.py` into\n   typed, concrete candidate-pair\n   manifests; the CUDA trainer batches those requests, the compiler absorbs the\n   dimensionless tournament ratios, and CV repeats before sealed evaluation.\n   The CUDA decode sweep trainer uses deterministic structurally-valid packed\n   payloads, not per-element random quantization, so giant LM-head refreshes do\n   not stall in host fixture generation. It still constructs the real tensor\n   classes and production VRAM-pool preparation path.\n   Policy fitting itself may use every CUDA and ROCm device through the exact\n   leaf-primary scorer DSOs. The wrapper auto-discovers them; use\n   `--policy-accelerators` and `--policy-lanes` to make ownership explicit.\n   These launches score authenticated regret matrices, not candidate kernels,\n   and cannot enter canonical timing or Nsight evidence. CUDA\u002FHIP workers are\n   process-isolated, and an explicitly requested accelerator failure is fatal.\n   A production refresh also emits a provenance-bound profiler request for\n   every physical candidate observation. Nsight Compute starts with collection\n   disabled; packing, graph capture, route proof, correctness, and warmup remain\n   outside profiling, and `cuProfilerStart` brackets one extra production\n   launch per pass. Never run `ncu` around canonical timing or import\n   replay-inflated profiler duration as the dispatch latency. Multi-kernel\n   candidates retain one ordered evidence record per physical dispatch.\n   Missing `ncu`, a required metric, or one supported candidate record blocks\n   a production install. The final `cuda_profiler_features.csv` is generated\n   only by authenticating the original canonical observation corpus against\n   the request and evidence manifests; it contains untouched pipeline timing\n   plus separate per-dispatch Nsight metrics.\n   The CUDA M1 decode family set is `wide,kpar,direct,fused_kpar`. The latter\n   retains the exact KPAR tree but publishes through CTA-local storage; its\n   resource admission and grouped inheritance belong to the common tuning\n   contract. The perf harness\n   uses the production VRAM-pool preparation path, which does not own ROWPAR's\n   optional row-major auxiliary weight view; do not add `rowpar` back to the\n   standard refresh unless the trainer has an explicit row-major-owner mode and\n   model-level parity proves the generated table.\n   If the full qwen36 inventory is too large for one pass, use the same staged\n   profiles as ROCm: `--profile qwen36-core` for FFN\u002FGDN projections and\n   `--profile qwen-mtp-head` for the complete MTP-specific matrix inventory.\n   The latter is not merely an LM-head alias: it resolves every release hidden\n   width into both the hidden\u002Fembedding projection `H x 2H` and terminal\n   `248320 x H` family, then covers M=1, grouped M=2..16, and the M31 sentinel\n   across every supported source format. Do not install either staged artifact\n   until the combined model-level parity and benchmark gates have passed.\n3. Use `--profile family-smoke` for a bounded representative training pass before\n   a full acceptance refresh. This profile is stratified by format: it runs one\n   small sweep per codebook\u002Ffamily, writes per-format partial CSVs, combines them,\n   then runs the normal train\u002Fgenerate\u002Fvalidate flow. This avoids a capped sweep\n   accidentally covering only the first format in the list.\n\n   ```bash\n   scripts\u002Frefresh_native_vnni_dispatch_tables.sh --backend cuda \\\n     --profile family-smoke \\\n     --cuda-formats Q4_0,IQ4_XS \\\n     --m-values 1,2\n   ```\n\n   `family-smoke` is a workflow\u002Fproxy gate, not production acceptance. Do not\n   replace broad checked-in tables from this profile alone; run `qwen36` or\n   `all`, then model-level parity and benchmark gates, before `--install`.\n   The wrapper intentionally uses proxy hit-rate thresholds for `family-smoke`;\n   production fallback-family\u002Fexact thresholds apply only to `qwen36` and `all`.\n\n4. Collect and install dense prefill exact overlays through the resumable\n   `native_vnni_dispatch.production_dense_prefill_sweep` transaction. It owns\n   the all-format Qwen geometry\u002FM inventory, one process per physical GPU,\n   per-candidate native-event timing sidecars, byte certificates, atomic cell\n   publication, resume validation, and deterministic combination. CUDA\n   candidates are only AUTO, direct BK64 output tiles, BK256, and the exact\n   public-M=1 K-partition tree; an independently chosen reduction tree is not a\n   launchable or benchmarkable candidate. Generate the include only after\n   `combine` authenticates every planned cell, using\n   `native_vnni_dispatch.dense_production_overlay`.\n5. When debugging the decode trainer itself, the lower-level flow is\n   `Perf__CUDANativeVNNIDecodeTrainer.cpp` ->\n   `analyze_cuda_native_vnni_decode_trainer.py` -> the common compiler under\n   `native_vnni_dispatch\u002F`. The retired `infer_gemv_dispatch_heuristic.py` and\n   `analyze_cuda_tc_gemv_dispatch.py` pipeline is not an authority and must not\n   be resurrected. Public M1 may legitimately prefer WIDE, DIRECT, or an exact\n   ordered KPAR schedule for the same `(N,K)` in eager versus captured mode.\n   Grouped M2+ never chooses an independent schedule; it inherits the complete\n   frozen M1 arithmetic identity and proves byte equality at every required M.\n   The trained fallback is the real policy: it must generalize by aspect ratio\n   and work-size\u002Flog-shape features. Exact `(M,N,K)` rows are overlays only.\n   Do not replace the broad fallback with a table that only recognizes today’s\n   model dimensions.\n   The overlay must score the runtime dispatch surface, not raw source-format\n   rows. CUDA NativeVNNI GEMV dispatch is keyed by `(codebook,M,N,K)`, and\n   aliases such as `Q4_1\u002FQ4_K`, `Q5_1\u002FQ5_K`, and `IQ4_NL\u002FIQ4_XS` cannot receive\n   separate runtime tunings for the same key. Collapse alias rows to one\n   aggregate codebook-level winner before enforcing exact-hit thresholds.\n   The trainer must bind an explicit non-blocking CUDA stream to the GEMM kernel\n   and record CUDA events on that stream. A `stream=0` trainer log or\n   `cudaEventRecord(start)` without a stream argument is a bug, not a benign perf\n   detail.\n   The trainer must prepare\u002Fupload\u002Frepack each weight once per format+shape\n   before candidate timing and size its `DeviceWorkspaceManager` from\n   `IWorkspaceConsumer::getWorkspaceRequirements()`; fixed 512 MiB budgets fail\n   on giant LM-head small-M partial buffers.\n6. Validate generated artifacts with the transaction unit suite,\n   `V2_Unit_CUDAGemvDispatchGeneratorAliases`, and\n   `V2_Unit_NativeVNNIGeneratedDispatchCodebooks`. For decode trainer changes,\n   include `V2_Unit_GpuWorkspaceAllocationPolicy` so the explicit-stream\n   trainer contract is checked. CUDA prefill installation additionally requires\n   `V2_Integration_CUDA_NativeVNNI_PrefillBucketMInvariantAllFormats` and the\n   backend GEMM source-policy gate; a timing winner without byte equality is an\n   invalid corpus row, not an installable exception.\n\nExample resumable CUDA prefill transaction:\n\n```bash\n# Preflight the complete compiler-resource surface before any long timing run.\nCUDA_VISIBLE_DEVICES=0 \\\nLLAMINAR_TILE_RESOURCE_CSV=\u002Ftmp\u002Fcuda-dense-prefill-resources.csv \\\nbuild_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm \\\n  --gtest_filter='CUDANativeVNNIGemmPerf.CompilerResources_AllCodebooksAndCandidates'\n\nPYTHONPATH=tests\u002Fv2\u002Fperformance\u002Fkernels python3 -m \\\n  native_vnni_dispatch.production_dense_prefill_sweep run \\\n  --backend cuda \\\n  --output-dir benchmark_results\u002Fnative_vnni_dispatch\u002Fwork\u002Fcuda-dense-prefill \\\n  --binary build_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm \\\n  --devices 0,1\n\nPYTHONPATH=tests\u002Fv2\u002Fperformance\u002Fkernels python3 -m \\\n  native_vnni_dispatch.production_dense_prefill_sweep combine \\\n  --backend cuda \\\n  --output-dir benchmark_results\u002Fnative_vnni_dispatch\u002Fwork\u002Fcuda-dense-prefill\n\nPYTHONPATH=tests\u002Fv2\u002Fperformance\u002Fkernels python3 -m \\\n  native_vnni_dispatch.dense_production_overlay \\\n  --backend cuda \\\n  --work-dir benchmark_results\u002Fnative_vnni_dispatch\u002Fwork\u002Fcuda-dense-prefill \\\n  --output src\u002Fv2\u002Fkernels\u002Fcuda\u002Fgemm\u002FCUDADenseProductionPrefillOverlayGenerated.inc \\\n  --summary-csv benchmark_results\u002Fnative_vnni_dispatch\u002Fwork\u002Fcuda-dense-prefill\u002Foverlay.csv\n```\n\nThe resource preflight enumerates every runtime codebook, all six BK64 tiles,\ndirect and canonical reduction, and both Q4_0 BK256 arithmetic forms. Its exact\nspill set is a regression. The scorer performs one untimed production-route\nprobe for AUTO, queries that exact primary\u002Freducer symbol with\n`cudaFuncGetAttributes`, and rejects nonzero local memory before recording any\ntiming event. Aggregate rows carry registers, static\u002Fdynamic shared memory,\nactual threads, compiler thread bounds, and active blocks\u002FSM; corpus validation\nauthenticates them. Never restore a spilling specialization to the candidate\ninventory merely because an older timing row was fast.\n\nAfter updating a checked-in generated include, rerun the focused CUDA GEMM route\nregression for the affected shape and the relevant Qwen3.6 CUDA parity cells.\n\n### MTP verifier dispatch mode\n\nGrouped MTP verifier rows are stricter than ordinary fast decode: rows may be\npublished to live state, so every native FP32 output byte must equal rowwise\nserial M1 decode. L2\u002Fcos\u002FKLD\u002Fmax-abs are diagnostics only. CUDA exposes this through\n`ITensorGemm::beginVerifierDecodeEquivalentScope()`, which selects the canonical\nsmall-M NativeVNNI dispatch\u002Freduction policy and disables prefill\u002Fconcurrent\ndecode reordering without enabling global `LLAMINAR_DETERMINISTIC`. Stage code\nmust use that shared RAII interface, never call CUDA `extern \"C\"` mode toggles\nor set environment variables directly.\n\n> **Build gotcha:** the MoE expert kernel `#include`s the decode header\n> `src\u002Fv2\u002Fkernels\u002Fcuda\u002Fgemm\u002FCUDANativeVNNIDecodeCommon.cuh`. After editing that header you\n> MUST `touch src\u002Fv2\u002Fkernels\u002Fcuda\u002Fmoe\u002FCUDAMoEKernels.cu` before `ninja`, or the change\n> won't be picked up. CUDA compiles are slow (~min, cicc-bound); ccache hits are fast.\n\n---\n\n## Step 5: Validate on the real model (the only verdict that counts)\n\nA perf-test win is necessary but **not sufficient**. Confirm on the full model and keep parity.\n\n```bash\n# 1. Rebuild engine\ntouch src\u002Fv2\u002Fkernels\u002Fcuda\u002Fmoe\u002FCUDAMoEKernels.cu   # if a .cuh include changed\nninja -C build_v2_release llaminar2\n\n# 2. Same-session A\u002FB (build baseline binary, measure; build variant, measure)\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m \u003Cmodel>.gguf -d cuda:0 2>\u002Fdev\u002Fnull | grep -iE \"Throughput\"\n\n# 3. Parity MUST stay PASS\nninja -C build_v2_integration llaminar2_core llaminar2\nctest --test-dir build_v2_integration -R \"\u003CModel>ParityTest_.*CUDA\" --output-on-failure\n```\n\nAccept the change only if: full-model throughput improves **beyond the noise band** AND\nparity stays PASS. Reject (and `git stash`\u002Frevert) otherwise.\n\n### Hard-won lesson (why isolated wins can regress the model)\n\nRegister-ceiling-bound fused kernels (e.g. the MoE prefill expert kernel: ~96 regs\u002Fthread,\n0 spills, ~92% of the register ceiling) trade throughput for occupancy. A decode\u002Fcodebook\nmicro-opt that is faster in isolation but **adds even a few temp registers** can drop\noccupancy in the fused kernel and *regress* full-model prefill, even though the standalone\nGEMM got faster. When the kernel is at the register ceiling, only **register-neutral** or\n**algorithmic** changes help — instruction-count tricks that cost registers will not.\n\n---\n\n## Step 6: Record the result\n\n- Production high-water marks live in `benchmarks\u002Fproduction\u002Fhigh_water.json`.\n  Only the official successful production pipeline commits those marks, after\n  both ISA images pass parity, the full E2E suites, and benchmarks. Use\n  `scripts\u002Fci\u002Frun_model_parity_benchmarks.py --diagnostic` for one-off experiments;\n  these reports cannot certify images or advance marks. See `docs\u002Fproduction-ci.md`.\n- Note rejected experiments and *why* (regression cause) so they aren't retried blindly.\n- Do **not** create ad-hoc markdown reports unless asked; use `changelog\u002F` (ISO-date prefix)\n  for durable write-ups.\n\n---\n\n## Quick reference — one-liners\n\n```bash\n# Hotspot ranking (relative only; decode runs eager under profiling)\nLLAMINAR_PERF_STATS_JSON=\u002Ftmp\u002Fcuda-profile.json \\\nLLAMINAR_PERF_STATS_GPU_STAGE_TIMING=1 \\\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m M.gguf -d cuda:0\n\n# Timeline + launch order\nsudo \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fnsys profile -t cuda --stats=true -o \u002Ftmp\u002Ft -f true \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 oneshot --no-mpi-bootstrap -d cuda:0 -m M.gguf -p \"x\" -n 10\n\n# Per-kernel counters (skip warmup, 1 launch)\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu --kernel-name \"K\" --launch-skip 1 --launch-count 1 \\\n  --section SpeedOfLight --section Occupancy --section LaunchStats \\\n  --section MemoryWorkloadAnalysis --section WarpStateStats --target-processes all \\\n  -o \u002Ftmp\u002Fk -f .\u002Fbuild_v2_release\u002Fllaminar2 oneshot --no-mpi-bootstrap -d cuda:0 -m M.gguf -p \"x\" -n 1\n\n# Cleanup (ALWAYS after ncu)\nsudo pkill -9 -f \"llaminar2 oneshot\"; sudo pkill -9 -f \"ncu --kernel\"; sleep 1; nvidia-smi\n\n# Scoped, correctness-gated GEMM A\u002FB\nLLAMINAR_CUDA_NATIVE_GEMM_SHAPES=\"ShapeA,ShapeB\" \\\n  .\u002Fbuild_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm\n\n# Clean-shell benchmark for the real number\nenv | grep -i LLAMINAR_PROFIL   # expect empty\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m M.gguf -d cuda:0 2>\u002Fdev\u002Fnull | grep -iE \"Throughput\"\n```\n","# CUDA Kernel Profiling & Tuning (Llaminar V2)\n\n## Purpose\n\nA repeatable, evidence-driven workflow for making Llaminar V2 CUDA kernels faster\nwithout breaking correctness. It chains four tools:\n\n1. **PerfStats GPU event timing** — production-graph replay timing and structured route\u002Fhost counters.\n2. **`nsys`** — timeline \u002F launch-count view to understand kernel ordering and CPU↔GPU overlap.\n3. **`ncu`** — per-kernel hardware-counter deep dive (occupancy, registers, spills, stalls).\n4. **Perf-test harness + benchmark + parity** — isolate, A\u002FB, and validate the change end-to-end.\n\nThe golden rule of this codebase: **isolated kernel speedups do not always translate\nto full-model speedups.** Always confirm a win on the real `benchmark` subcommand, and\nalways re-run parity before trusting it.\n\n## Production hot-path invariants\n\nCUDA kernel tuning must preserve the production execution architecture, not\ntrade architecture for an isolated microbenchmark win:\n\n- **No dynamic allocation or deallocation.** Hot-path kernels and launch\n  bridges consume persistent workspace bindings. `cudaMalloc`,\n  `cudaMallocAsync`, `cudaFree`, container growth, or hidden scratch allocation\n  belongs in explicit initialization\u002Fworkspace infrastructure only.\n- **No host transfer.** Do not insert H2D\u002FD2H copies, host-visible mirrors, or\n  host polling into execution. Deliberate RAM\u002Fdisk KV-cache tier movement is a\n  transfer-service operation at an explicit lifecycle boundary, not kernel\n  plumbing.\n- **No stream or device synchronization.** `cudaStreamSynchronize`,\n  `cudaDeviceSynchronize`, blocking copies, and event synchronization on the\n  host are forbidden in the hot path. They are permitted only in explicitly\n  scoped test\u002Fprofiler result collection or terminal result surfacing.\n- **No order-dependent atomics in batch-invariant paths.** Grouped verifier,\n  deterministic decode, and batch-invariant prefill kernels must give every\n  output a unique deterministic writer. An atomic is acceptable only when its\n  value is provably independent of inter-thread order, bitwise equivalence is\n  covered by the all-format\u002FM-totality sweep, and profiling proves it is\n  economical.\n- **Use explicit producer\u002Fconsumer events.** A producer launches on an explicit\n  non-null stream and publishes completion by recording an event on that exact\n  stream. A consumer on another stream waits for that event before reading.\n  Never publish a write on a guessed\u002Fdefault stream, transition coherence\n  manually after a blocking sync, or replace the dependency with a global\n  barrier.\n- **Remain graph-capturable end to end.** Persistent buffers, launches, event\n  record\u002Fwait edges, and collectives must be capturable. Homogeneous GPU\n  inference requires one complete captured graph; segmented execution is not a\n  tuning fallback.\n\n---\n\n## Step 0: Establish the target and baseline\n\nBefore touching anything, capture a baseline so every change is measured against it.\n\n```bash\n# Release build of the engine (always Ninja, never limit parallelism)\nninja -C build_v2_release llaminar2\n\n# Baseline throughput (1st \"Throughput\" line = prefill, 2nd = decode)\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m \u003Cmodel>.gguf -d cuda:0 2>\u002Fdev\u002Fnull | grep -iE \"Throughput\"\n```\n\nRecord the prefill\u002Fdecode tok\u002Fs and the noise band (run 2-3×; typical noise is a few\ntok\u002Fs on prefill, \u003C0.5 tok\u002Fs on decode). A change inside the noise band is **not** a win.\n\n> **Parity gate (must stay PASS the whole time):**\n> ```bash\n> ctest --test-dir build_v2_integration -R \"\u003CModel>ParityTest_(Prefill|Decode)Parity.*CUDA\" --output-on-failure\n> ```\n\n---\n\n## Step 1: Find the hotspot with graph-safe PerfStats\n\nPerfStats measures the same production graph topology used for ordinary\ninference. Enable structured output and explicit GPU-event timing around graph\nreplay:\n\n```bash\nLLAMINAR_PERF_STATS_JSON=\u002Ftmp\u002Fcuda-profile.json \\\nLLAMINAR_PERF_STATS_SUMMARY=1 \\\nLLAMINAR_PERF_STATS_GPU_STAGE_TIMING=1 \\\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m \u003Cmodel>.gguf -d cuda:0\n```\n\nRank `stage_gpu.graph_replay.total` and graph\u002Fsegment rows alongside\n`forward_graph`, `forward_pass`, `kernel`, `mtp`, and model-specific route\ncounters. A monolithic captured graph intentionally does not pretend to have\nper-stage event attribution inside replay. Use `nsys` to identify kernels\ninside that graph, then target a single launch with `ncu`.\n\n`LLAMINAR_PROFILING=1` is deprecated. It aliases the PerfStats summary and GPU\nreplay-event request for compatibility, emits a warning, and must never disable\ncapture or select eager execution. New commands and automation must use the\nexplicit `LLAMINAR_PERF_STATS_*` variables.\n\nPick the single highest-time stage that is plausibly tunable (skip stages that are\nalready compute-bound at the math limit, e.g. the big dense GEMMs, unless that's the\nexplicit target).\n\n---\n\n## Profiler attachment and privilege rules\n\n- Resolve Nsight Systems with `command -v nsys`; the devcontainer package\n  installs it as `\u002Fusr\u002Flocal\u002Fbin\u002Fnsys`, independently of the CUDA toolkit.\n  Nsight Compute remains `\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu`.\n- Run both tools through `sudo -E` on hosts where\n  `\u002Fproc\u002Fdriver\u002Fnvidia\u002Fparams` reports `RmProfilingAdminOnly: 1`. An\n  unprivileged `nsys` launch can appear successful while collecting only CUDA\n  graph-construction metadata and no executed kernels.\n- Preserve `LLAMINAR_*` variables across privilege escalation with `sudo -E`,\n  or pass the exact variables after `sudo`. Ordinary `sudo` strips them.\n- Pass `--no-mpi-bootstrap` only when profiling or debugging `llaminar2`\n  directly; otherwise Nsight attaches to the `mpirun` wrapper. Never use the\n  flag for production or canonical benchmark measurements.\n- Do not pass `--no-mpi-bootstrap` to standalone test\u002Fperformance binaries;\n  they do not auto-bootstrap MPI.\n- Write `.nsys-rep` and `.ncu-rep` artifacts under `\u002Ftmp` or another explicit\n  result directory, not in the repository.\n\n### Prove profiler attachment before a model run\n\nDo not spend a model load on an unproven profiler setup. First profile a focused\nintegration test that executes one eager launch and one captured replay:\n\n```bash\nsudo -E \"$(command -v nsys)\" profile \\\n  --trace=cuda,nvtx --sample=none --cpuctxsw=none \\\n  --cuda-graph-trace=node --force-overwrite=true \\\n  --output=\u002Ftmp\u002Fllaminar-nsys-cuda-smoke \\\n  .\u002Fbuild_v2_integration\u002Ftests\u002Fv2\u002Fv2_integration_cuda_moe_kernel \\\n  --gtest_filter=Test__CUDAMoEKernel.RouteWithTensorsTiledPrefillCapturesAfterWarmup\n\n\"$(command -v nsys)\" stats --report cuda_gpu_kern_sum,cuda_api_sum \\\n  \u002Ftmp\u002Fllaminar-nsys-cuda-smoke.nsys-rep\n```\n\nThe smoke report must contain a `CUDA GPU Kernel Summary`, including both\nlaunch instances from the focused test. A report containing\n`CUDA_GRAPH_NODE_EVENTS` or `CUDA_GRAPH_EVENTS` but no\n`CUPTI_ACTIVITY_KIND_KERNEL` is **not** an execution trace. Its diagnostics\nnormally say `CUDA profiling might have not been started correctly` or report\nonly graph-node creation. Check privilege first.\n\nCUDA 13 conditional\u002Fdevice-loop graphs expose a second tool boundary on Ampere:\nthe tested Nsight Systems 2025.3 and 2026.1 releases can collect ordinary eager\nand simple captured graphs yet omit all activities from Llaminar's complete\nproduction conditional graph. Confirm this by comparing the smoke test with the\nreal graph report; never disable production graph execution and present the\neager timeline as production evidence. Use targeted Nsight Compute node\nprofiling below for the real graph.\n\nDo not defer `ncu` attachment with `--profile-from-start off` or a late\n`cudaProfilerStart()` range for this graph topology. That technique profiles a\nsimple already-instantiated graph, but the tested Llaminar conditional graph\nexposes no inner kernel nodes when profiling begins after graph construction.\nKeep NCU active from process start and use its graph-node kernel filter to\nselect one production launch.\n\n---\n\n## Step 2: Timeline sanity check with `nsys`\n\nUse `nsys` to confirm the kernel of interest dominates and to see launch counts \u002F sync stalls.\n\n```bash\n# --no-mpi-bootstrap is REQUIRED so the profiler attaches to llaminar2, not the mpirun wrapper\nsudo -E \"$(command -v nsys)\" profile -t cuda --stats=true -o \u002Ftmp\u002Ftrace -f true \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 oneshot --no-mpi-bootstrap -d cuda:0 \\\n  -m \u003Cmodel>.gguf -p \"test\" -n 10\n\n# Per-kernel summary (gives the launch order needed for ncu --launch-skip)\n\"$(command -v nsys)\" stats --report cuda_gpu_kern_sum \u002Ftmp\u002Ftrace.nsys-rep\n```\n\n`nsys` also tells you the kernel's mangled name and how many times it launches — both\nneeded to target `ncu` precisely.\n\n> `--no-mpi-bootstrap` is ONLY for profiler\u002Fdebugger attach (`ncu`\u002F`nsys`\u002F`gdb`\u002F`perf`).\n> It disables NUMA-aware thread pinning, so it gives misleading *performance* numbers —\n> never use it for benchmarks or production.\n\n### Inventory durations inside the complete production graph with NCU\n\nConditional\u002Fdevice-loop graphs on Ampere may be invisible to Nsight Systems\neven when a simple graph smoke is visible. Use this proven one-counter NCU\ntransaction to enumerate the real production graph before selecting a deep\nprofile. The critical details are: attach to `llaminar2` directly with\n`--no-mpi-bootstrap`, keep profiling active from process start, request graph\n**node** profiling, use application replay, and filter out model-load kernels\nso the launch budget reaches captured inference:\n\n```bash\nRUNTIME_KERNELS='regex:(buildGrouped|build_active|count_per|cuda_attention|cuda_derive_attention|cuda_gated|cuda_gdn|cuda_q_gate|cuda_short_conv1d|exclusive_scan|float_to_int|fp32_|grouped_|groupedImma|quantize_activations|router_gate|scatter_tokens|shared_expert|softmax_topk|embedding_lookup|fused_|mtpConcat|requestTerminal|cuda_kv|residual_add|rmsnorm|rope_|flash_attention|nativeVnniTC|route_logits|shiftedMTP|ring_append|ring_gather)'\n\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --target-processes all \\\n  --replay-mode application \\\n  --graph-profiling node \\\n  --kernel-name-base demangled \\\n  --kernel-name \"$RUNTIME_KERNELS\" \\\n  --launch-count 1200 \\\n  --metrics gpu__time_duration.sum \\\n  --clock-control none \\\n  --force-overwrite \\\n  --export \u002Ftmp\u002Fllaminar-production-node-duration \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 benchmark --no-mpi-bootstrap \\\n    -m \u003Cmodel.gguf> -d cuda:0 \\\n    --prompt-file \u003Cfixed-prompt.txt> \\\n    --seed \u003Cseed> --temperature \u003Ctemperature> \\\n    --mtp --mtp-draft-tokens \u003Cdepth> \\\n    --mtp-depth-policy fixed \\\n    --mtp-verify-mode speculative-sampling \\\n    \u003Cremaining-exact-release-arguments>\n\n\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --import \u002Ftmp\u002Fllaminar-production-node-duration.ncu-rep \\\n  --page raw --csv \\\n  > \u002Ftmp\u002Fllaminar-production-node-duration.csv\n\n# Recover the exact profiler command and target provenance from any report.\n\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --import \u002Ftmp\u002Fllaminar-production-node-duration.ncu-rep \\\n  --page session --csv\n```\n\nChoose `--launch-count` above the expected runtime-node count, then verify the\nCSV reached the terminal graph kernels rather than silently truncating at the\nbudget. Sum `gpu__time_duration.sum` by exact demangled kernel name and retain\ncall counts plus grid\u002Fblock geometry; aggregate names without geometry can\nhide one pathological shape behind many cheap launches. Inspect the CSV units\nrow before summing: NCU auto-scales `gpu__time_duration.sum` and may emit `us`\nfor one report and `ms` for another. Normalize every row to one unit rather than\nassuming a fixed scale. This first pass uses one metric and is for attribution\nonly. It does not replace warmed unprofiled Release timing, and it does not\ncertify occupancy or spills.\n\nIf NCU prints `No kernels were profiled`, first run the profiler-attachment\nsmoke above, then inspect the report's `Available Kernels`. Do not switch to\neager execution, segmented graphs, late `cudaProfilerStart()`, or a host-side\nsurrogate to make the profiler easier to use; those are different execution\npaths.\n\n---\n\n## Step 3: Deep-dive a kernel with `ncu`\n\n`ncu` replays each kernel 4-8× through hardware counters, so always **target one kernel**\nand **skip warmups**.\n\n```bash\n# sudo -E preserves env (LLAMINAR_*); ncu needs sudo for counter access\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --kernel-name \"\u003Cmangled_or_substr>\" \\\n  --launch-skip 1 --launch-count 1 \\\n  --section SpeedOfLight \\\n  --section Occupancy \\\n  --section LaunchStats \\\n  --section MemoryWorkloadAnalysis \\\n  --section WarpStateStats \\\n  --section ComputeWorkloadAnalysis \\\n  --target-processes all \\\n  -o \u002Ftmp\u002Fk_ncu -f \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 oneshot --no-mpi-bootstrap -d cuda:0 \\\n  -m \u003Cmodel>.gguf -p \"test\" -n 1\n\n# Read it back\nsudo \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu -i \u002Ftmp\u002Fk_ncu.ncu-rep --page details\n\n# Inspect the sections supported by the installed Nsight version\n\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu --list-sections\n```\n\n### Profile a node inside the real captured graph\n\n`ncu` has a graph-node profiler independent of the Nsight Systems timeline.\nUse it when the Systems version cannot expand Llaminar's conditional graph:\n\n```bash\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --target-processes all --graph-profiling node \\\n  --kernel-name-base demangled \\\n  --kernel-name 'regex:\u003Cunique-production-kernel-substring>' \\\n  --launch-count 1 \\\n  --section LaunchStats --section Occupancy --section SpeedOfLight \\\n  --section MemoryWorkloadAnalysis \\\n  --clock-control none --force-overwrite \\\n  --export \u002Ftmp\u002Fllaminar-real-node \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 benchmark --no-mpi-bootstrap \u003Cexact-release-args>\n\nsudo \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --import \u002Ftmp\u002Fllaminar-real-node.ncu-rep \\\n  --page details --print-details all\n```\n\nMatch one unique kernel family and set `--launch-count 1`. Default kernel replay\nbacks up device-written memory before each metric pass; with a resident 35B\nmodel this can take roughly a minute even for one node. Keep the section set\nminimal, or collect a one-pass custom metric list when only spill\u002Fvectorization\nproof is needed. Never interpret the profiled duration as canonical latency;\nuse the unprofiled Release benchmark for timing.\n\nKernel families and template winners can change after rebuilding generated\ndispatch. If a selector produces `No kernels were profiled`, read NCU's\n`Available Kernels` list and select the exact current production family. A\ndeliberately impossible selector with a one-pass section such as `LaunchStats`\nis a quick inventory probe. Do not carry yesterday's selector into today's\nprofile without checking it.\n\nFor the Qwen 3.6 35B grouped-prefill path, a proven real-graph selector is\n`regex:groupedImmaProjectionKernel`. `MemoryWorkloadAnalysis` must report zero\n`derived__local_spilling_requests`; `LaunchStats`, `Occupancy`, and\n`SpeedOfLight` supply registers, launch geometry, achieved occupancy, and the\ncompute\u002Fmemory throughput balance.\n\n> 🧹 **MANDATORY CLEANUP after every ncu run** (ncu leaves zombie processes that hold\n> the GPU and corrupt the next run):\n> ```bash\n> sudo pkill -9 -f \"llaminar2 oneshot\"; sudo pkill -9 -f \"ncu --kernel\"; sleep 1; nvidia-smi\n> ```\n\n### What to read first\n\n| Metric | Healthy | Red flag → likely cause |\n|--------|---------|-------------------------|\n| `Local Memory Spilling` | 0 | >0 → register pressure spilling to DRAM |\n| `Registers Per Thread` | matches `__launch_bounds__` | unexpected → check launch config \u002F occupancy ceiling |\n| `Compute (SM) Throughput` | >60% (compute-bound) | \u003C30% → latency-bound |\n| `DRAM Throughput` | \u003C30% (compute-bound) | >60% w\u002F low compute → spilling or bad access pattern |\n| `Achieved Occupancy` | near theoretical | low + low compute → not enough warps to hide latency |\n| `Warp Cycles Per Issued Instruction` | \u003C15 | >30 → severe stalls (check WarpStateStats breakdown) |\n\n| Warp stall dominated by | Cause | Fix direction |\n|-------------------------|-------|---------------|\n| `Barrier` | uneven work across `__syncthreads()` | fewer sync points, warp-level sync |\n| `L1\u002FTEX` + spilling | spilled regs reloaded from local mem | reduce live registers \u002F relax `__launch_bounds__` MIN_BLOCKS |\n| `Long Scoreboard` (memory) | global-load latency | improve coalescing \u002F prefetch \u002F more warps |\n\n### Target standalone and A\u002FB launches\n\nRun `nsys` first and count only launches matching the eventual `--kernel-name`\nfilter. `--launch-skip` counts matching launches, not all launches in the\nprocess. Profile variant A and variant B in separate, otherwise-identical\ninvocations with `--launch-count 1`; do not compare replay duration from the\nreports. Use the saved reports only for counters and resource diagnostics, and\nuse the unprofiled harness for canonical latency.\n\nFor standalone performance binaries, invoke `nsys`\u002F`ncu` on the binary\ndirectly. Keep `--target-processes all` when the harness can fork, preserve its\nshape\u002Fformat\u002FM filters, and recalculate launch skip whenever the warmup or\ncandidate order changes.\n\n### Prove NativeVNNI IMMA and vectorized memory execution\n\nDo not infer tensor-core use from a source-level `mma.sync` spelling or a\ncandidate-family name. Profile the actual production-selected candidate and\nrecord the GA102 IMMA and memory-width counters on one isolated extra launch:\n\n```bash\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu \\\n  --kernel-name \"nativeVnniTC\" --launch-skip 1 --launch-count 1 \\\n  --metrics \\\nsm__inst_executed_pipe_tensor_op_imma.sum,\\\nsmsp__average_inst_executed_pipe_tensor_op_imma_per_warp,\\\nsmsp__sass_inst_executed_op_memory_128b.sum,\\\nsmsp__sass_average_data_bytes_per_sector_mem_global_op_ld.ratio \\\n  --target-processes all -o \u002Ftmp\u002Fnative_vnni_instruction_proof -f \\\n  .\u002Fbuild_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm \\\n  --gtest_filter=\"CUDANativeVNNIGemmPerf.Performance_AllFormats_AllShapes\"\n```\n\nThe command MUST use the harness's normal shape\u002Fformat\u002FM filters so Nsight sees\nthe candidate selected by production dispatch. A nonzero IMMA count proves the\ncompiled integer tensor-core path. The 128-bit instruction count and bytes per\nsector diagnose vectorization and coalescing independently. Nsight replay time\nis never canonical latency and must not enter dispatch labels.\n\n---\n\n## Step 4: Isolate & A\u002FB with the GEMM perf-test harness\n\nFor GEMM\u002FMoE-expert kernels, the perf test gives a fast, correctness-gated A\u002FB loop that\nis far quicker than full-model rebuilds. It shares the production decode\u002Fcodebook helpers,\nso it is a valid proxy for those kernels.\n\n```bash\n# Build the perf test (Release)\nninja -C build_v2_release tests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm\n\n# Correctness gate is cosine >= 0.9990 vs cuBLAS. The full-shape sweep is SLOW\n# (single-threaded CPU reference, several minutes) — ALWAYS scope to the shapes you care about.\nLLAMINAR_CUDA_NATIVE_GEMM_SHAPES=\"35BMoE_Expert_GateUp,35BMoE_Expert_Down\" \\\n  .\u002Fbuild_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm --gtest_filter=\"*\"\n```\n\nUseful env filters (comma-separated, case-insensitive): `LLAMINAR_CUDA_NATIVE_GEMM_FORMATS`,\n`LLAMINAR_CUDA_NATIVE_GEMM_SHAPES`, `LLAMINAR_CUDA_NATIVE_GEMM_PREFILL_M`,\n`LLAMINAR_CUDA_NATIVE_GEMM_BENCH`, `LLAMINAR_CUDA_NATIVE_GEMM_WORKERS`.\n\nAdd new production shapes to `kQwenShapes` in\n`tests\u002Fv2\u002Fperformance\u002Fkernels\u002Fcuda\u002Fgemm\u002FCUDANativeVNNIGemmPerfCommon.h` so the harness\ncovers the exact GEMMs a model actually runs.\n\n### Generated GEMM\u002FGEMV dispatch training pipeline\n\nRead `.agents\u002Fnativevnni-gemm-tuning\u002FSKILL.md` for the shared corpus, learner,\ncertification, Git LFS, and installation workflow. This section owns only\nCUDA-specific kernel\u002Fprofiler constraints.\n\nDo not land source-level \"one shape gets this tile\" overrides for CUDA NativeVNNI\nunless the user explicitly asks for a temporary experiment. The durable path is:\n\n1. Add or confirm the production shapes in\n   `tests\u002Fv2\u002Fperformance\u002Fkernels\u002Fcuda\u002Fgemm\u002FCUDANativeVNNIGemmPerfCommon.h`.\n2. For decode\u002FGEMV dispatch tables, use the Git-LFS-aware turnkey transaction:\n   `scripts\u002Ftrain_native_vnni_dispatch.sh --backend cuda --install`.\n   The lower-level `refresh_native_vnni_dispatch_tables.sh` remains the phase\n   executor used by the driver and for focused diagnostic collection.\n   It first sweeps and freezes `Fast M=1`, then certifies grouped verifier\n   `M=2..16,31` against that exact staged serial policy. Fifteen drafts require\n   verifier M16 because the target pass includes one bonus row; M31 is a deeper\n   sentinel, not a runtime maximum. Installation requires this complete matrix.\n   The M1 broad sweep runs complete sample-interleaved candidate rounds. Failed\n   development-CV cells are converted by\n   `tests\u002Fv2\u002Fperformance\u002Fkernels\u002Fnative_vnni_dispatch\u002Fpaired_requests.py` into\n   typed, concrete candidate-pair\n   manifests; the CUDA trainer batches those requests, the compiler absorbs the\n   dimensionless tournament ratios, and CV repeats before sealed evaluation.\n   The CUDA decode sweep trainer uses deterministic structurally-valid packed\n   payloads, not per-element random quantization, so giant LM-head refreshes do\n   not stall in host fixture generation. It still constructs the real tensor\n   classes and production VRAM-pool preparation path.\n   Policy fitting itself may use every CUDA and ROCm device through the exact\n   leaf-primary scorer DSOs. The wrapper auto-discovers them; use\n   `--policy-accelerators` and `--policy-lanes` to make ownership explicit.\n   These launches score authenticated regret matrices, not candidate kernels,\n   and cannot enter canonical timing or Nsight evidence. CUDA\u002FHIP workers are\n   process-isolated, and an explicitly requested accelerator failure is fatal.\n   A production refresh also emits a provenance-bound profiler request for\n   every physical candidate observation. Nsight Compute starts with collection\n   disabled; packing, graph capture, route proof, correctness, and warmup remain\n   outside profiling, and `cuProfilerStart` brackets one extra production\n   launch per pass. Never run `ncu` around canonical timing or import\n   replay-inflated profiler duration as the dispatch latency. Multi-kernel\n   candidates retain one ordered evidence record per physical dispatch.\n   Missing `ncu`, a required metric, or one supported candidate record blocks\n   a production install. The final `cuda_profiler_features.csv` is generated\n   only by authenticating the original canonical observation corpus against\n   the request and evidence manifests; it contains untouched pipeline timing\n   plus separate per-dispatch Nsight metrics.\n   The CUDA M1 decode family set is `wide,kpar,direct,fused_kpar`. The latter\n   retains the exact KPAR tree but publishes through CTA-local storage; its\n   resource admission and grouped inheritance belong to the common tuning\n   contract. The perf harness\n   uses the production VRAM-pool preparation path, which does not own ROWPAR's\n   optional row-major auxiliary weight view; do not add `rowpar` back to the\n   standard refresh unless the trainer has an explicit row-major-owner mode and\n   model-level parity proves the generated table.\n   If the full qwen36 inventory is too large for one pass, use the same staged\n   profiles as ROCm: `--profile qwen36-core` for FFN\u002FGDN projections and\n   `--profile qwen-mtp-head` for the complete MTP-specific matrix inventory.\n   The latter is not merely an LM-head alias: it resolves every release hidden\n   width into both the hidden\u002Fembedding projection `H x 2H` and terminal\n   `248320 x H` family, then covers M=1, grouped M=2..16, and the M31 sentinel\n   across every supported source format. Do not install either staged artifact\n   until the combined model-level parity and benchmark gates have passed.\n3. Use `--profile family-smoke` for a bounded representative training pass before\n   a full acceptance refresh. This profile is stratified by format: it runs one\n   small sweep per codebook\u002Ffamily, writes per-format partial CSVs, combines them,\n   then runs the normal train\u002Fgenerate\u002Fvalidate flow. This avoids a capped sweep\n   accidentally covering only the first format in the list.\n\n   ```bash\n   scripts\u002Frefresh_native_vnni_dispatch_tables.sh --backend cuda \\\n     --profile family-smoke \\\n     --cuda-formats Q4_0,IQ4_XS \\\n     --m-values 1,2\n   ```\n\n   `family-smoke` is a workflow\u002Fproxy gate, not production acceptance. Do not\n   replace broad checked-in tables from this profile alone; run `qwen36` or\n   `all`, then model-level parity and benchmark gates, before `--install`.\n   The wrapper intentionally uses proxy hit-rate thresholds for `family-smoke`;\n   production fallback-family\u002Fexact thresholds apply only to `qwen36` and `all`.\n\n4. Collect and install dense prefill exact overlays through the resumable\n   `native_vnni_dispatch.production_dense_prefill_sweep` transaction. It owns\n   the all-format Qwen geometry\u002FM inventory, one process per physical GPU,\n   per-candidate native-event timing sidecars, byte certificates, atomic cell\n   publication, resume validation, and deterministic combination. CUDA\n   candidates are only AUTO, direct BK64 output tiles, BK256, and the exact\n   public-M=1 K-partition tree; an independently chosen reduction tree is not a\n   launchable or benchmarkable candidate. Generate the include only after\n   `combine` authenticates every planned cell, using\n   `native_vnni_dispatch.dense_production_overlay`.\n5. When debugging the decode trainer itself, the lower-level flow is\n   `Perf__CUDANativeVNNIDecodeTrainer.cpp` ->\n   `analyze_cuda_native_vnni_decode_trainer.py` -> the common compiler under\n   `native_vnni_dispatch\u002F`. The retired `infer_gemv_dispatch_heuristic.py` and\n   `analyze_cuda_tc_gemv_dispatch.py` pipeline is not an authority and must not\n   be resurrected. Public M1 may legitimately prefer WIDE, DIRECT, or an exact\n   ordered KPAR schedule for the same `(N,K)` in eager versus captured mode.\n   Grouped M2+ never chooses an independent schedule; it inherits the complete\n   frozen M1 arithmetic identity and proves byte equality at every required M.\n   The trained fallback is the real policy: it must generalize by aspect ratio\n   and work-size\u002Flog-shape features. Exact `(M,N,K)` rows are overlays only.\n   Do not replace the broad fallback with a table that only recognizes today’s\n   model dimensions.\n   The overlay must score the runtime dispatch surface, not raw source-format\n   rows. CUDA NativeVNNI GEMV dispatch is keyed by `(codebook,M,N,K)`, and\n   aliases such as `Q4_1\u002FQ4_K`, `Q5_1\u002FQ5_K`, and `IQ4_NL\u002FIQ4_XS` cannot receive\n   separate runtime tunings for the same key. Collapse alias rows to one\n   aggregate codebook-level winner before enforcing exact-hit thresholds.\n   The trainer must bind an explicit non-blocking CUDA stream to the GEMM kernel\n   and record CUDA events on that stream. A `stream=0` trainer log or\n   `cudaEventRecord(start)` without a stream argument is a bug, not a benign perf\n   detail.\n   The trainer must prepare\u002Fupload\u002Frepack each weight once per format+shape\n   before candidate timing and size its `DeviceWorkspaceManager` from\n   `IWorkspaceConsumer::getWorkspaceRequirements()`; fixed 512 MiB budgets fail\n   on giant LM-head small-M partial buffers.\n6. Validate generated artifacts with the transaction unit suite,\n   `V2_Unit_CUDAGemvDispatchGeneratorAliases`, and\n   `V2_Unit_NativeVNNIGeneratedDispatchCodebooks`. For decode trainer changes,\n   include `V2_Unit_GpuWorkspaceAllocationPolicy` so the explicit-stream\n   trainer contract is checked. CUDA prefill installation additionally requires\n   `V2_Integration_CUDA_NativeVNNI_PrefillBucketMInvariantAllFormats` and the\n   backend GEMM source-policy gate; a timing winner without byte equality is an\n   invalid corpus row, not an installable exception.\n\nExample resumable CUDA prefill transaction:\n\n```bash\n# Preflight the complete compiler-resource surface before any long timing run.\nCUDA_VISIBLE_DEVICES=0 \\\nLLAMINAR_TILE_RESOURCE_CSV=\u002Ftmp\u002Fcuda-dense-prefill-resources.csv \\\nbuild_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm \\\n  --gtest_filter='CUDANativeVNNIGemmPerf.CompilerResources_AllCodebooksAndCandidates'\n\nPYTHONPATH=tests\u002Fv2\u002Fperformance\u002Fkernels python3 -m \\\n  native_vnni_dispatch.production_dense_prefill_sweep run \\\n  --backend cuda \\\n  --output-dir benchmark_results\u002Fnative_vnni_dispatch\u002Fwork\u002Fcuda-dense-prefill \\\n  --binary build_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm \\\n  --devices 0,1\n\nPYTHONPATH=tests\u002Fv2\u002Fperformance\u002Fkernels python3 -m \\\n  native_vnni_dispatch.production_dense_prefill_sweep combine \\\n  --backend cuda \\\n  --output-dir benchmark_results\u002Fnative_vnni_dispatch\u002Fwork\u002Fcuda-dense-prefill\n\nPYTHONPATH=tests\u002Fv2\u002Fperformance\u002Fkernels python3 -m \\\n  native_vnni_dispatch.dense_production_overlay \\\n  --backend cuda \\\n  --work-dir benchmark_results\u002Fnative_vnni_dispatch\u002Fwork\u002Fcuda-dense-prefill \\\n  --output src\u002Fv2\u002Fkernels\u002Fcuda\u002Fgemm\u002FCUDADenseProductionPrefillOverlayGenerated.inc \\\n  --summary-csv benchmark_results\u002Fnative_vnni_dispatch\u002Fwork\u002Fcuda-dense-prefill\u002Foverlay.csv\n```\n\nThe resource preflight enumerates every runtime codebook, all six BK64 tiles,\ndirect and canonical reduction, and both Q4_0 BK256 arithmetic forms. Its exact\nspill set is a regression. The scorer performs one untimed production-route\nprobe for AUTO, queries that exact primary\u002Freducer symbol with\n`cudaFuncGetAttributes`, and rejects nonzero local memory before recording any\ntiming event. Aggregate rows carry registers, static\u002Fdynamic shared memory,\nactual threads, compiler thread bounds, and active blocks\u002FSM; corpus validation\nauthenticates them. Never restore a spilling specialization to the candidate\ninventory merely because an older timing row was fast.\n\nAfter updating a checked-in generated include, rerun the focused CUDA GEMM route\nregression for the affected shape and the relevant Qwen3.6 CUDA parity cells.\n\n### MTP verifier dispatch mode\n\nGrouped MTP verifier rows are stricter than ordinary fast decode: rows may be\npublished to live state, so every native FP32 output byte must equal rowwise\nserial M1 decode. L2\u002Fcos\u002FKLD\u002Fmax-abs are diagnostics only. CUDA exposes this through\n`ITensorGemm::beginVerifierDecodeEquivalentScope()`, which selects the canonical\nsmall-M NativeVNNI dispatch\u002Freduction policy and disables prefill\u002Fconcurrent\ndecode reordering without enabling global `LLAMINAR_DETERMINISTIC`. Stage code\nmust use that shared RAII interface, never call CUDA `extern \"C\"` mode toggles\nor set environment variables directly.\n\n> **Build gotcha:** the MoE expert kernel `#include`s the decode header\n> `src\u002Fv2\u002Fkernels\u002Fcuda\u002Fgemm\u002FCUDANativeVNNIDecodeCommon.cuh`. After editing that header you\n> MUST `touch src\u002Fv2\u002Fkernels\u002Fcuda\u002Fmoe\u002FCUDAMoEKernels.cu` before `ninja`, or the change\n> won't be picked up. CUDA compiles are slow (~min, cicc-bound); ccache hits are fast.\n\n---\n\n## Step 5: Validate on the real model (the only verdict that counts)\n\nA perf-test win is necessary but **not sufficient**. Confirm on the full model and keep parity.\n\n```bash\n# 1. Rebuild engine\ntouch src\u002Fv2\u002Fkernels\u002Fcuda\u002Fmoe\u002FCUDAMoEKernels.cu   # if a .cuh include changed\nninja -C build_v2_release llaminar2\n\n# 2. Same-session A\u002FB (build baseline binary, measure; build variant, measure)\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m \u003Cmodel>.gguf -d cuda:0 2>\u002Fdev\u002Fnull | grep -iE \"Throughput\"\n\n# 3. Parity MUST stay PASS\nninja -C build_v2_integration llaminar2_core llaminar2\nctest --test-dir build_v2_integration -R \"\u003CModel>ParityTest_.*CUDA\" --output-on-failure\n```\n\nAccept the change only if: full-model throughput improves **beyond the noise band** AND\nparity stays PASS. Reject (and `git stash`\u002Frevert) otherwise.\n\n### Hard-won lesson (why isolated wins can regress the model)\n\nRegister-ceiling-bound fused kernels (e.g. the MoE prefill expert kernel: ~96 regs\u002Fthread,\n0 spills, ~92% of the register ceiling) trade throughput for occupancy. A decode\u002Fcodebook\nmicro-opt that is faster in isolation but **adds even a few temp registers** can drop\noccupancy in the fused kernel and *regress* full-model prefill, even though the standalone\nGEMM got faster. When the kernel is at the register ceiling, only **register-neutral** or\n**algorithmic** changes help — instruction-count tricks that cost registers will not.\n\n---\n\n## Step 6: Record the result\n\n- Production high-water marks live in `benchmarks\u002Fproduction\u002Fhigh_water.json`.\n  Only the official successful production pipeline commits those marks, after\n  both ISA images pass parity, the full E2E suites, and benchmarks. Use\n  `scripts\u002Fci\u002Frun_model_parity_benchmarks.py --diagnostic` for one-off experiments;\n  these reports cannot certify images or advance marks. See `docs\u002Fproduction-ci.md`.\n- Note rejected experiments and *why* (regression cause) so they aren't retried blindly.\n- Do **not** create ad-hoc markdown reports unless asked; use `changelog\u002F` (ISO-date prefix)\n  for durable write-ups.\n\n---\n\n## Quick reference — one-liners\n\n```bash\n# Hotspot ranking (relative only; decode runs eager under profiling)\nLLAMINAR_PERF_STATS_JSON=\u002Ftmp\u002Fcuda-profile.json \\\nLLAMINAR_PERF_STATS_GPU_STAGE_TIMING=1 \\\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m M.gguf -d cuda:0\n\n# Timeline + launch order\nsudo \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fnsys profile -t cuda --stats=true -o \u002Ftmp\u002Ft -f true \\\n  .\u002Fbuild_v2_release\u002Fllaminar2 oneshot --no-mpi-bootstrap -d cuda:0 -m M.gguf -p \"x\" -n 10\n\n# Per-kernel counters (skip warmup, 1 launch)\nsudo -E \u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fncu --kernel-name \"K\" --launch-skip 1 --launch-count 1 \\\n  --section SpeedOfLight --section Occupancy --section LaunchStats \\\n  --section MemoryWorkloadAnalysis --section WarpStateStats --target-processes all \\\n  -o \u002Ftmp\u002Fk -f .\u002Fbuild_v2_release\u002Fllaminar2 oneshot --no-mpi-bootstrap -d cuda:0 -m M.gguf -p \"x\" -n 1\n\n# Cleanup (ALWAYS after ncu)\nsudo pkill -9 -f \"llaminar2 oneshot\"; sudo pkill -9 -f \"ncu --kernel\"; sleep 1; nvidia-smi\n\n# Scoped, correctness-gated GEMM A\u002FB\nLLAMINAR_CUDA_NATIVE_GEMM_SHAPES=\"ShapeA,ShapeB\" \\\n  .\u002Fbuild_v2_release\u002Ftests\u002Fv2\u002Fv2_perf_cuda_native_vnni_gemm\n\n# Clean-shell benchmark for the real number\nenv | grep -i LLAMINAR_PROFIL   # expect empty\n.\u002Fbuild_v2_release\u002Fllaminar2 benchmark -m M.gguf -d cuda:0 2>\u002Fdev\u002Fnull | grep -iE \"Throughput\"\n```\n",{"name":5,"description":7},"master","454741b1b2228bc6874e46ade75bbf523a8385d5",".agents\u002Fcuda-tuning",4,"https:\u002F\u002Fgithub.com\u002FLlaminar\u002Fllaminar","https:\u002F\u002Fllaminar.ai","https:\u002F\u002Fgithub.com\u002FLlaminar","Home of the Llaminar inferencing engine.","2026-11-15 04:04:16",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add Llaminar\u002Fllaminar --skill cuda-kernel-profiling",{"tool":58,"command":59},"bunx","bunx skills add Llaminar\u002Fllaminar --skill cuda-kernel-profiling",{"tool":61,"command":62},"pnpm","pnpm dlx skills add Llaminar\u002Fllaminar --skill cuda-kernel-profiling","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FLlaminar\u002Fllaminar\u002Fblob\u002Fmaster\u002F.agents\u002Fcuda-tuning\u002FSKILL.md",true,[],[67,76,84,89,97,102],{"skillNo":68,"name":69,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":38},"01M718P9ZZSG781K5PXM1V3KG2","rocm-tuning","https:\u002F\u002Fgithub.com\u002FLlaminar\u002Fllaminar\u002Fblob\u002Fmaster\u002F.codex\u002Fskills\u002Frocm-tuning\u002FSKILL.md",[],[73,74,75],"rocm tuning","tuning","rocm",{"skillNo":77,"name":78,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":38},"01M718P95WX88GYN9MTSC6AE40","mtp-tuning","https:\u002F\u002Fgithub.com\u002FLlaminar\u002Fllaminar\u002Fblob\u002Fmaster\u002F.codex\u002Fskills\u002Fmtp-tuning\u002FSKILL.md",[],[82,74,83],"mtp tuning","mtp",{"skillNo":85,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":38},"01M718P7JK26A3N6VZYSE2J36T","https:\u002F\u002Fgithub.com\u002FLlaminar\u002Fllaminar\u002Fblob\u002Fmaster\u002F.codex\u002Fskills\u002Fcuda-kernel-profiling\u002FSKILL.md",[],[19,18,20,21],{"skillNo":90,"name":91,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":38},"01M718P5YCB9J40NZZ0PRAKS5V","cpu-tuning","https:\u002F\u002Fgithub.com\u002FLlaminar\u002Fllaminar\u002Fblob\u002Fmaster\u002F.codex\u002Fskills\u002Fcpu-tuning\u002FSKILL.md",[],[95,96,74],"cpu tuning","CPU",{"skillNo":98,"name":69,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":38},"01M718P57PZ4G3X1B24ZVWR2PQ","https:\u002F\u002Fgithub.com\u002FLlaminar\u002Fllaminar\u002Fblob\u002Fmaster\u002F.claude\u002Fskills\u002Frocm-tuning\u002FSKILL.md",[],[73,74,75],{"skillNo":103,"name":104,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":38},"01M718P4GF2X3EHBTY7ZB1Q2T2","nativevnni-gemm-tuning","https:\u002F\u002Fgithub.com\u002FLlaminar\u002Fllaminar\u002Fblob\u002Fmaster\u002F.claude\u002Fskills\u002Fnativevnni-gemm-tuning\u002FSKILL.md",[],[74,108],"nativevnni gemm tuning"]