gfx-deep-learning-neural-rendering-and-performance
Use when applying learned methods or profiling a pipeline: deep learning for vision including detection, segmentation, backbones and the training realities; neural rendering where graphics and vision meet, covering NeRF, 3D Gaussian splatting and their trade-offs; and performance — profiling, bottleneck analysis, and the memory and bandwidth limits that dominate in practice.
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 5
- Forks
- 3
01 前言 Front Matter
- name
- gfx-deep-learning-neural-rendering-and-performance
- description
- Use when applying learned methods or profiling a pipeline: deep learning for vision including detection, segmentation, backbones and the training realities; neural rendering where graphics and vision meet, covering NeRF, 3D Gaussian splatting and their trade-offs; and performance — profiling, bottleneck analysis, and the memory and bandwidth limits that dominate in practice.
03 SKILL.md 正文
Graphics and Vision: Deep Learning for Vision, Neural Rendering, and Performance
Part 4 of 5 of the Graphics and Vision reference (plugin
graphics-and-vision), covering §11–§13. Sibling skills:gfx-transforms-rasterization-and-rendering(§0–§4),gfx-gpu-real-time-techniques-and-colour(§5–§7),gfx-image-formation-classical-vision-and-geometry(§8–§10),gfx-reference(§14–§19). Section numbers are shared across the set; a reference written as §N →skillpoints into that sibling skill.Currency: The mathematics is permanent — projective geometry, the rendering equation, epipolar geometry. Neural rendering and vision foundation models moved. See §16 →
gfx-referencefor those.
Why one document. ⚠️ They are inverse problems over the same geometry. Graphics goes scene → image; vision goes image → scene. They share the camera model, projective geometry, and increasingly the representations — §12 is where they now meet directly.
⚠️ GOTCHA boxes mark where the math is subtle or the standard implementation is silently wrong.
The three ideas that unify the whole document:
- ⚠️ Homogeneous coordinates make projection linear. Adding a fourth component turns perspective — a nonlinear operation — into a matrix multiply, which is why the entire pipeline is matrices (§1 →
gfx-transforms-rasterization-and-rendering).- ⚠️ Rendering is an integral, and everything real-time is a sampling strategy for it. The rendering equation is the ground truth; rasterization, path tracing, and every approximation in §6 →
gfx-gpu-real-time-techniques-and-colourare ways of estimating it under a time budget (§4 →gfx-transforms-rasterization-and-rendering).- ⚠️ Vision is underdetermined and that is not a solvable bug. A single image is a projection; depth is destroyed. Every vision method is adding a constraint — multiple views, motion, a learned prior — to recover what the projection threw away (§8 →
gfx-image-formation-classical-vision-and-geometry, §10 →gfx-image-formation-classical-vision-and-geometry).
§11. Deep Learning for Vision
⚠️ The general ML framework sits in an ML reference; here's what's vision-specific.
CNNs: convolution as a learned filter bank with weight sharing and translation equivariance — ⚠️ the inductive bias that made vision learnable with limited data. Receptive field, stride, dilation, pooling. ResNet's skip connections solved the degradation problem and made depth trainable.
Vision Transformers: image → patches → tokens → self-attention. ⚠️ ViTs have weaker inductive bias than CNNs, so they need more data or stronger augmentation — but they scale better and dominate at large scale. Hierarchical variants (Swin) reintroduce locality.
Tasks and the standard architectures: classification; detection (two-stage
R-CNN family, one-stage YOLO/SSD/RetinaNet, ⚠️ DETR's set-prediction formulation removed
NMS and anchor design — a genuine simplification); segmentation (semantic: U-Net,
DeepLab with atrous convolution; instance: Mask R-CNN; panoptic); depth estimation; pose
estimation; tracking; and generation (§16.2 → gfx-reference for the current model landscape).
⚠️ The practical failure modes that matter more than architecture choice:
- Data quality and label noise dominate. ⚠️ Almost always worth more than a better model.
- Augmentation is where much of the performance lives — and ⚠️ an augmentation that breaks the task's invariance (horizontal flip on text or on left/right-labelled data) silently caps your ceiling.
- Class imbalance — focal loss, resampling.
- ⚠️ Domain shift: train on daylight, deploy at night. Test-set performance is not deployment performance, and the gap is where vision systems fail in the field.
- ⚠️ Shortcut learning — the model keys on the watermark, the hospital's scanner, or the ruler in the frame. See a biomedical-engineering reference §4.3 for documented cases.
§12. Neural Rendering — Where the Fields Meet
⚠️ The problem: novel view synthesis. Given photos of a scene, render it from a viewpoint you never captured. This is vision (recover the scene) and graphics (render it) as a single optimization, and it's the most significant development in either field in a decade.
NeRF (2020) — represent the scene as a continuous function (x, y, z, θ, φ) → (colour, density), learned as an MLP, rendered by volumetric ray marching, optimized
by comparing rendered pixels to captured photos. Differentiable rendering is the key
idea — the renderer is the loss function's forward pass.
⚠️ NeRF's problem was always cost: volumetric rendering requires many network
evaluations per ray. Instant-NGP's multiresolution hash encoding cut training to minutes;
rendering stayed slow.
3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023) — represent the scene as millions of anisotropic 3D Gaussian ellipsoids with position, covariance, opacity, and view-dependent colour via spherical harmonics. Initialize from SfM points, then interleave optimization with adaptive density control (splitting and cloning Gaussians where reconstruction error is high), and rasterize with a fast visibility-aware differentiable splatting algorithm.
⚠️ The conceptual point that makes 3DGS click: it is explicit, not neural. ⚠️ There is no network evaluated at render time at all — it's a rasterization of primitives, which is why it hits real-time on a GPU that was already built to rasterize. The "neural" part is the optimization, not the representation. ⚠️ It's also not new in lineage — EWA splatting dates to 2002; what changed is differentiable optimization and adaptive densification.
§16.1 → gfx-reference for the current state.
⚠️ Limitations worth knowing regardless of version: specular and reflective surfaces are handled poorly (spherical harmonics can't represent view-dependent reflection well, and the optimizer compensates by scattering Gaussians, hurting geometry); extracting a clean mesh from either representation is a separate, imperfect step; relighting is mostly unsolved because appearance and illumination are entangled; and memory/storage for high-fidelity scenes is large, which is why compression is such an active area.
§13. Performance
Graphics: ⚠️ profile before optimizing, and identify which stage is the bottleneck — vertex, fragment, memory bandwidth, or CPU submission. Tools: RenderDoc, Nsight Graphics, PIX, Xcode's frame debugger, RGP. Common wins: reduce draw calls (instancing, batching, ⚠️ indirect and bindless rendering), LOD, culling, ⚠️ overdraw reduction via depth prepass or front-to-back sorting, texture compression and mipmaps, and shader complexity in that order. ⚠️ The frame budget is brutal: 16.6 ms at 60 fps, 8.3 ms at 120, ~11 ms per eye at 90 for VR — and VR's cost of a dropped frame is nausea, not a stutter.
Vision: ⚠️ the biggest wins are usually resolution and preprocessing, not model architecture. Then quantization (INT8), pruning, distillation, batching, and the right runtime (TensorRT, ONNX Runtime, OpenVINO, Core ML, TFLite). ⚠️ Measure end-to-end including decode, resize and colour conversion — on many pipelines the model is not the bottleneck.
04 同仓库更多 Skills
white-paper-writing
Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks "how should I structure this" for a long-form document.
prose-technical-mechanics
Use when editing or writing prose at a technical/mechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.
prose-mechanics
Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate/Jamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix "AI-sounding" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.
narrative-structure
Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or "how should I structure this story/pitch/presentation" request.
legal-brief-writing
Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic/neurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.
journal-publishability-criteria
Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs/paper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.