AGENT SKILL

dsp-audio-rf-and-images

Use when working in a specific signal domain: audio and speech (psychoacoustics, coding and codecs, the real-time voice pipeline with echo cancellation, noise suppression and latency budgets, music and analysis), RF and communications (modulation, software-defined radio, synchronization, equalization), and images and video treated as two-dimensional signals.

License 许可
MIT
最近更新
2026-11-15
Stars
★ 5
Forks
3

01 前言 Front Matter

name
dsp-audio-rf-and-images
description
Use when working in a specific signal domain: audio and speech (psychoacoustics, coding and codecs, the real-time voice pipeline with echo cancellation, noise suppression and latency budgets, music and analysis), RF and communications (modulation, software-defined radio, synchronization, equalization), and images and video treated as two-dimensional signals.

03 SKILL.md 正文

Signal Processing: Audio and Speech, RF and Communications, and Images and Video

Part 3 of 5 of the Signal Processing reference (plugin signal-processing), covering §8–§10. Sibling skills: dsp-sampling-frequency-domain-and-filters (§0–§3), dsp-convolution-multirate-and-spectral-analysis (§4–§7), dsp-implementation-tools-and-testing (§11–§14), dsp-reference (§15–§20). Section numbers are shared across the set; a reference written as §N → skill points into that sibling skill.

Currency: Verified August 2026. See §17 → dsp-reference for the currency snapshot and what goes stale first.

How to read this. Reference for engineers who need DSP working, not a course. Three markers:

  • [DURABLE] — sampling theory, transforms, filters, estimation. Nyquist is 1928; Shannon 1949; the Cooley–Tukey FFT 1965. This material does not expire.
  • [VERSIONED] — codecs, libraries, hardware, the ML layer.
  • [CONTESTED] — genuine disagreement.

⚠️ GOTCHA boxes mark where the code runs, the plot looks fine, and the answer is wrong.

The three framings that organize everything below:

  1. Everything is a trade between time and frequency resolution, and you cannot cheat it. The uncertainty principle is not a metaphor here — a short window gives you good time resolution and poor frequency resolution, and no amount of cleverness escapes that (§2.4 → dsp-sampling-frequency-domain-and-filters). Most DSP design is choosing where on that curve to sit.
  2. ⚠️ Aliasing is irreversible. Once you've sampled too slowly, the information is gone — no filter, no algorithm, and no neural network recovers it. This is the one mistake in this document that cannot be fixed downstream (§1 → dsp-sampling-frequency-domain-and-filters).
  3. The field is mid-convergence with machine learning, and the interesting result is that hybrids are winning — not pure DSP, and not pure end-to-end learning. ⚠️ The pattern that keeps recurring: use DSP for the structure you know, and learn the parameters you can't hand-tune (§8, §16.1 → dsp-reference).

§8. Audio and Speech

8.1 The domain facts

[DURABLE] Human hearing spans roughly 20 Hz – 20 kHz (⚠️ and the top end declines substantially with age), with ~120 dB of dynamic range, logarithmic in both frequency and amplitude. Critical bands / Bark / ERB scales model the ear's frequency resolution, and masking — a loud tone hiding a nearby quiet one, in frequency and in time — is the entire basis of perceptual audio coding.

⚠️ Latency budgets are what constrain audio engineering: musicians notice round-trip latency above roughly 10 ms; conversational latency becomes awkward above ~150–200 ms one-way. These numbers determine your block size, and block size determines everything else.

8.2 Coding

Perceptual codecs (MP3, AAC, Vorbis, Opus) throw away what masking says you can't hear: MDCT transform, psychoacoustic model, quantize, entropy-code. Speech codecs (⚠️ model the vocal tract rather than the waveform): LPC, CELP, AMR, and Opus's SILK layer.

[VERSIONED] Opus is the one to know — RFC 6716, hybrid SILK + CELT, mandatory in WebRTC, and covers everything from speech at low bitrate to full-band music.

⚠️ GOTCHA — and the most interesting story in DSP right now: classical codecs are absorbing neural components rather than being replaced by them.

Opus 1.5 (March 2024) put machine learning inside the codec for the first time, all at the decoder — meaning a service can adopt it on the playback side alone and improve calls for users on bad networks without changing the sender.

  • Deep PLC — a neural network reconstructs lost packets, in place of the classical repeat-and-fade. Recommended for occasional loss.
  • DRED (Deep REDundancy) — for burst loss. It uses a rate-distortion-optimized VAE to compress acoustic features so aggressively that up to one second of redundancy fits in ~12–32 kb/s of overhead, carried in each packet's padding. ⚠️ It transmits 20 acoustic features — 18 Bark-frequency cepstral coefficients plus pitch and voicing — and resynthesizes speech from them. Note the consequence: the features contain no phase information, so the recovered waveform deviates significantly from the original while sounding similar.
  • LACE/NoLACE — a DNN post-filter that denoises the decoder's output.

Opus 1.6 (Dec 2025) added bandwidth extension (BWE), which combined with DRED reaches fullband quality, and with NoLACE enables good fullband speech as low as 9 kb/s.

⚠️ Two caveats worth carrying. DRED's format was still being finalised in IETF as of 2026 — treat it as advanced rather than settled. And any DNN in a codec significantly increases real-time compute, which is the actual adoption barrier — along with model size, which Opus's own developers name as a main obstacle.

Neural codecs proper — SoundStream, EnCodec, and the discrete-token codecs underpinning generative audio — are a genuinely different lineage: much lower bitrates, much higher compute, and tokens that double as inputs to generative models.

8.3 The real-time voice pipeline

[DURABLE] The order matters and is roughly fixed: AEC → noise suppression → dereverberation → AGC → codec.

Acoustic echo cancellation — ⚠️ adaptive filter (§6 → dsp-convolution-multirate-and-spectral-analysis) to model the echo path, plus a nonlinear residual echo suppressor, plus a double-talk detector. Interestingly, most state-of-the-art AEC is still classical DSP or hybrid DSP-ML, with deep learning typically replacing only the nonlinear residual suppressor.

Noise suppression — and this is where the convergence story is clearest. The classical line runs spectral subtraction (Boll) → ⚠️ which produced the notorious "musical noise" artifact → Wiener filtering (needs a priori statistics) → MMSE log-spectral amplitude (Ephraim–Malah, 1985), still a respectable baseline.

RNNoise (Valin, 2018) is the landmark hybrid: a small recurrent network estimates ideal critical-band gains while a traditional pitch filter attenuates noise between harmonics — ⚠️ significantly better than a classical MMSE estimator while staying real-time at 48 kHz on a low-power CPU. DeepFilterNet and successors extended this by predicting linear filters rather than estimating clean speech directly, which is computationally far cheaper and edge-deployable.

⚠️ The honest current comparison: traditional approaches are fast but conservative; deep learning can be more aggressive while preserving speech quality, at higher computational cost. In hearing-aid studies, deep and classical methods perform similarly in diffuse noise, with the deep binaural approach winning in the presence of spatial interferers. The pattern throughout: hybrids, not replacements.

Beamforming for multi-mic arrays — delay-and-sum, MVDR, GSC — ⚠️ spatial filtering that is genuinely complementary to spectral methods, and increasingly paired with neural mask estimation.

8.4 Music and analysis

Pitch detection (autocorrelation, YIN, CREPE), onset detection, beat tracking, source separation (⚠️ Demucs, Spleeter and the neural separators substantially beat classical NMF-based methods — one of the cleanest ML wins in audio), MFCCs (⚠️ the classical speech feature; largely superseded by learned representations and mel spectrograms in modern ML pipelines, but still everywhere in legacy code), time-scale and pitch modification (phase vocoder, PSOLA, WSOLA).


§9. RF and Communications

Modulation: analogue (AM/FM/PM), digital (ASK/FSK/PSK/QAM), and OFDM — ⚠️ which converts a frequency-selective channel into many flat sub-channels and is why it's in Wi-Fi, LTE, 5G, and DVB. Spread spectrum (DSSS, FHSS) for interference resistance and multiple access.

The receive chain: down-conversion to I/Q baseband (⚠️ complex-valued representation is the whole language of RF DSP — get comfortable with it), matched filtering (§4 → dsp-convolution-multirate-and-spectral-analysis), timing and carrier recovery, equalization (§6 → dsp-convolution-multirate-and-spectral-analysis), and demodulation.

Error correction: Hamming and Reed–Solomon (⚠️ still the standard for burst errors in storage and broadcast), convolutional codes with Viterbi decoding, Turbo, LDPC (⚠️ near-Shannon-limit, and in 5G, Wi-Fi 6 and DVB-S2), and polar codes in 5G control channels.

⚠️ The physical realities that dominate: multipath and fading, Doppler, the noise figure of your front end, and the fact that Shannon capacity is a hard ceiling — C = B log₂(1 + SNR). No modulation scheme beats it.

SDR is where software engineers actually meet this: GNU Radio (flowgraph-based), RTL-SDR (⚠️ ~$30 and genuinely capable for receive-only experimentation), HackRF, USRP, LimeSDR, and SoapySDR as the hardware abstraction. ⚠️ Transmitting requires a licence in most jurisdictions and most bands — receiving generally doesn't.


§10. Images and Video

[DURABLE] 2D signal processing, and the concepts transfer directly. 2D convolution for blur, sharpen, and edge detection (Sobel, Laplacian); separable kernels (⚠️ a 2D Gaussian is two 1D passes — O(n) instead of O(n²) per pixel, and this is why Gaussian blur is cheap); 2D FFT for frequency-domain filtering; the DCT as JPEG's core.

⚠️ Aliasing appears here as moiré patterns — and image downsampling without pre-filtering is exactly §1.1 → dsp-sampling-frequency-domain-and-filters's error in two dimensions. Anti-aliasing in rendering is the same problem approached from the synthesis side.

Also: image pyramids (Gaussian, Laplacian) for multi-scale processing; bilateral and non-local means filters (⚠️ edge-preserving — linear filters can't do this); optical flow for motion; video codecs as motion-compensated transform coding (H.264/AVC, H.265/HEVC, AV1, and H.266/VVC).

04 同仓库更多 Skills

the-vibey-project
the-vibey-project/vibey

white-paper-writing

Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks "how should I structure this" for a long-form document.

★ 5
the-vibey-project
the-vibey-project/vibey

prose-technical-mechanics

Use when editing or writing prose at a technical/mechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.

★ 5
the-vibey-project
the-vibey-project/vibey

prose-mechanics

Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate/Jamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix "AI-sounding" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.

★ 5
the-vibey-project
the-vibey-project/vibey

narrative-structure

Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or "how should I structure this story/pitch/presentation" request.

★ 5
the-vibey-project
the-vibey-project/vibey

legal-brief-writing

Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic/neurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.

★ 5
the-vibey-project
the-vibey-project/vibey

journal-publishability-criteria

Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs/paper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.

★ 5