uarch-gpu-npu-dataflow-and-numeric-formats
Use for accelerators: GPU microarchitecture including warps, occupancy and divergence, GPU memory systems with coalescing and the bandwidth hierarchy, NPUs and dataflow architectures such as systolic arrays and why they suit tensor workloads, and numeric formats from FP32 down to FP8 and the block-scaled formats with their accuracy and throughput trade-offs.
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 5
- Forks
- 3
01 前言 Front Matter
- name
- uarch-gpu-npu-dataflow-and-numeric-formats
- description
- Use for accelerators: GPU microarchitecture including warps, occupancy and divergence, GPU memory systems with coalescing and the bandwidth hierarchy, NPUs and dataflow architectures such as systolic arrays and why they suit tensor workloads, and numeric formats from FP32 down to FP8 and the block-scaled formats with their accuracy and throughput trade-offs.
03 SKILL.md 正文
Microarchitecture and Memory: GPU Microarchitecture, GPU Memory Systems, NPUs and Dataflow Architectures, and Numeric Formats
Part 3 of 6 of the CPU, GPU, NPU and Memory Microarchitecture reference (plugin
microarchitecture-and-memory-systems), covering §11–§14. Sibling skills:uarch-pipelining-out-of-order-branch-prediction-and-simd(§0–§5),uarch-caches-coherence-consistency-and-virtual-memory(§6–§10),uarch-dram-memory-controllers-power-and-security(§15–§19),uarch-isa-simulation-measurement-roofline-and-specialization(§20–§24),uarch-reference(§25–§30). Section numbers are shared across the set; a reference written as §N →skillpoints into that sibling skill.Currency: The principles are stable. Two areas moved. See §25 →
uarch-referencefor low-precision numeric formats, and next-generation memory standards.
⚠️ SCOPE, because this sits between two neighbours. ⚠️ A semiconductor reference covers device physics and fabrication — how a transistor works and how it's made. A computer-hardware reference covers system integration — components, builds, facilities. ⚠️ THIS file is the layer in between: how billions of transistors are ORGANIZED into something that executes instructions fast, and why memory systems are the hard part. ⚠️ It assumes the other two rather than repeating them.
⚠️ GOTCHA boxes mark where architectural intuition is wrong and where the abstraction leaks.
The three ideas that organize this document:
- ⚠️ THE PROCESSOR LIES TO YOU, DELIBERATELY (§2–§4 →
uarch-pipelining-out-of-order-branch-prediction-and-simd). Instructions do not execute in program order, memory operations do not become visible in program order, and the architecture works hard to maintain the illusion that they do. Every performance surprise and several security disasters live in the gap.- ⚠️ MOVING DATA COSTS FAR MORE THAN COMPUTING ON IT (§6 →
uarch-caches-coherence-consistency-and-virtual-memory, §13, §23 →uarch-isa-simulation-measurement-roofline-and-specialization). Energy per arithmetic operation has fallen enormously; energy per byte moved has not. Modern architecture is organized around minimizing data movement, and that single fact explains caches, dataflow accelerators, quantization and HBM alike.- ⚠️ PARALLELISM IS EXTRACTED AT EVERY LEVEL, AND EACH HAS A DIFFERENT LIMIT (§3 →
uarch-pipelining-out-of-order-branch-prediction-and-simd, §11, §14). ILP is limited by dependencies and branches, DLP by divergence and memory, TLP by synchronization and coherence traffic. Knowing which one you're hitting is the entire diagnostic skill.
§11. GPU Microarchitecture
⚠️ THE ORGANIZING PRINCIPLE ⚠️ hide latency with PARALLELISM
rather than with caches and speculation. ⚠️ When a warp stalls
on memory, the scheduler switches to another — so latency is
tolerated rather than reduced
⚠️ THE HIERARCHY ⚠️ SM / CU containing warp schedulers, register
file, SIMD lanes, ⚠️ SHARED MEMORY / LDS (⚠️ software-managed
scratchpad — the key GPU-specific resource), L1, tensor units
⚠️ SIMT ⚠️ threads grouped into warps (32) or wavefronts (32/64)
executing in lockstep
⚠️ WARP DIVERGENCE — ⚠️ if threads in a warp take different
branch paths, the hardware SERIALIZES them. ⚠️ Worst case a
32-way divergent branch costs 32× — the single biggest GPU
performance trap
⚠️ ⚠️ OCCUPANCY active warps per SM, limited by ⚠️ REGISTERS PER
THREAD, ⚠️ SHARED MEMORY PER BLOCK, and block size.
⚠️ GOTCHA: higher occupancy is NOT automatically better —
beyond the point where latency is hidden, more warps just
thrash cache. ⚠️ Register-heavy kernels with low occupancy
frequently outperform "optimized" high-occupancy versions
⚠️ MEMORY COALESCING ⚠️ consecutive threads accessing consecutive
addresses combine into one transaction; ⚠️ scattered access
multiplies the transaction count and is the second big trap
⚠️ BANK CONFLICTS in shared memory ⚠️ serialize access
⚠️ THE REGISTER FILE IS HUGE ⚠️ larger than the L1 cache — because
thousands of resident threads each need private state
§12. GPU Memory Systems
⚠️ GDDR versus HBM: ⚠️ GDDR is cheaper with fewer, faster pins; ⚠️ HBM uses stacked
dies and an extremely wide interface (§15 → uarch-dram-memory-controllers-power-and-security, and see a semiconductor reference §20) for
far higher bandwidth per watt at much higher cost.
⚠️ Bandwidth is the design centre: ⚠️ GPUs are usually bandwidth-bound (§23 → uarch-isa-simulation-measurement-roofline-and-specialization), so
arithmetic intensity determines whether you can use the compute at all.
⚠️ Caches on GPUs behave differently: ⚠️ smaller per thread, and used more for
coalescing and reuse across warps than for latency reduction.
⚠️ Unified/managed memory simplifies programming and ⚠️ can hide expensive migration
— convenience that costs performance silently.
⚠️ Multi-GPU: ⚠️ NVLink and Infinity Fabric provide far higher bandwidth than PCIe,
and collective operations make the interconnect part of the compute path (see a
computer-hardware reference §21).
§13. ⚠️ NPUs and Dataflow Architectures
⚠️ The least-covered area elsewhere, and the design logic is genuinely different from both CPU and GPU.
⚠️ THE PREMISE ⚠️ neural network inference and training are
dominated by MATRIX MULTIPLY, which has enormous, REGULAR,
STATICALLY KNOWN parallelism. ⚠️ You do not need speculation,
branch prediction or coherence — so remove them and spend
the area on arithmetic and local memory
⚠️ ⚠️ THE ENERGY ARGUMENT IS THE REAL ONE ⚠️ a multiply-accumulate
costs far less energy than fetching its operands from DRAM.
⚠️ Therefore the architecture is designed around DATA REUSE,
not around arithmetic throughput
⚠️ SYSTOLIC ARRAYS ⚠️ a grid of MAC units where data flows
rhythmically between neighbours — ⚠️ each value fetched once
and reused across the array. Google's TPU is the canonical
modern example
⚠️ DATAFLOW TAXONOMY ⚠️ weight-stationary (keep weights in place,
stream activations) · output-stationary (accumulate in place) ·
row-stationary. ⚠️ The choice depends on layer shape, and
flexible accelerators support several
⚠️ THE MEMORY HIERARCHY IS EXPLICIT AND SOFTWARE-MANAGED —
⚠️ no transparent caches. ⚠️ TILING and scheduling are the
compiler's job, and the compiler is most of the product
⚠️ SPARSITY ⚠️ structured (e.g. 2:4) is hardware-exploitable;
unstructured sparsity is much harder to convert into speedup
⚠️ COMPUTE-IN-MEMORY research direction — do the MAC where the
data lives, attacking the movement cost directly
⚠️ EDGE NPUs ⚠️ optimized for INT8 and now INT4/FP4 (§25.1),
with fixed function blocks and very tight power budgets
§14. ⚠️ Numeric Formats
⚠️ Precision is an architectural parameter now, not a given.
⚠️ IEEE 754 FP64, FP32, ⚠️ FP16 (5 exponent bits — limited RANGE,
which causes training overflow)
⚠️ BF16 ⚠️ FP32's 8 exponent bits with fewer mantissa bits —
⚠️ same RANGE as FP32, less precision. ⚠️ This is why BF16
largely displaced FP16 for training: range mattered more
than mantissa
⚠️ FP8 E4M3 (more precision) and E5M2 (more range) — typically
used together for forward and backward passes
⚠️ INTEGER INT8, INT4 — with quantization scale and zero-point
⚠️ ⚠️ BLOCK FLOATING POINT / MICROSCALING ⚠️ the key modern idea:
a group of low-precision values SHARES a higher-precision
scale factor, recovering dynamic range at almost no bit cost
(§25.1)
⚠️ THE TRADE ⚠️ halving bit width roughly halves memory, memory
BANDWIDTH and energy, and can more than double throughput —
⚠️ which is why quantization is the highest-leverage
optimization available for inference
⚠️ WHAT BREAKS ⚠️ ACTIVATION OUTLIERS — a few very large values
force a scale that crushes everything else. ⚠️ Hence
per-channel and per-block scaling, and Hadamard rotations
that spread outlier energy across dimensions
⚠️ ⚠️ NOT EVERYTHING QUANTIZES softmax, layer norm and
accumulation are typically kept at higher precision
PART III — MEMORY SYSTEMS
04 同仓库更多 Skills
white-paper-writing
Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks "how should I structure this" for a long-form document.
prose-technical-mechanics
Use when editing or writing prose at a technical/mechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.
prose-mechanics
Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate/Jamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix "AI-sounding" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.
narrative-structure
Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or "how should I structure this story/pitch/presentation" request.
legal-brief-writing
Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic/neurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.
journal-publishability-criteria
Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs/paper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.