stream-pipeline-patterns
暂无简介
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 13
- Forks
- 2
03 SKILL.md 正文
/stream-pipeline-patterns
When to Use
- Chaining compress → encrypt → write streams with proper error cleanup
- Processing large log files or archives without loading into memory
- Ensuring all streams in a pipeline are destroyed if any one errors
- Replacing manual
.pipe()chains that leak on errors
Do NOT use for
- One-shot small data (< 10MB) — just use
fs.readFileSync/Buffer - Async iterables (use
for await...ofwith stream iteration instead)
pump (auto-cleanup on error)
import pump from 'pump'
import { createReadStream, createWriteStream } from 'fs'
import { createGzip } from 'zlib'
// If any stream errors, ALL streams are destroyed and callback fires
pump(
createReadStream('/tmp/agent.log'),
createGzip(),
createWriteStream('/tmp/agent.log.gz'),
(err) => {
if (err) console.error('[pump] pipeline error:', err.message)
else console.log('[pump] done')
}
)
stream.pipeline (built-in Node.js ≥ 10)
import { pipeline } from 'stream/promises'
import { createReadStream, createWriteStream } from 'fs'
import { createBrotliCompress } from 'zlib'
// Awaitable pipeline — throws on error, cleans up all streams
async function compressLog(input: string, output: string): Promise<void> {
await pipeline(
createReadStream(input),
createBrotliCompress(),
createWriteStream(output)
)
}
Transform stream chain
import { Transform } from 'stream'
import { pipeline } from 'stream/promises'
const lineCounter = new Transform({
transform(chunk, _enc, cb) {
const lines = chunk.toString().split('\n').length - 1
this.lineCount = (this.lineCount ?? 0) + lines
cb(null, chunk)
},
})
const jsonFilter = new Transform({
transform(chunk, _enc, cb) {
const filtered = chunk.toString()
.split('\n')
.filter(line => { try { JSON.parse(line); return true } catch { return false } })
.join('\n')
cb(null, filtered)
},
})
await pipeline(
createReadStream('/tmp/audit.log'),
jsonFilter,
lineCounter,
createWriteStream('/tmp/audit-clean.log')
)
Backpressure: respect highWaterMark
const readable = createReadStream('/tmp/huge.log', {
highWaterMark: 64 * 1024, // 64KB chunks
})
// pipeline handles backpressure automatically
// manual .pipe() requires checking write.write() return value
Anti-Fake-Pass Checklist
❌ Manual .pipe() without error handlers → source stream not destroyed on dest error
❌ pump callback-style — must handle null err case, not just error case
❌ stream.pipeline (non-promise) vs stream/promises — different import paths
❌ Transform pushes data after calling cb(err) → double-emit crash
❌ highWaterMark too large → buffers entire file in memory despite streaming intent
❌ pipeline with async generator → must use Node.js 16+ for full support
04 同仓库更多 Skills
openai--sentry--sentry
Use when the user asks to inspect Sentry issues or events, summarize recent production errors, or pull basic Sentry health data via the Sentry API; perform read-only queries with the bundled script and require `SENTRY_AUTH_TOKEN`.
ito-basket-compare
Compare Itô prediction-market baskets against a user's knowledge base, portfolio notes, financial context, watchlist, or research thesis. Use for read-only basket comparison and gap analysis without investment advice or live trading.
implementing-network-traffic-baselining
Build network traffic baselines from NetFlow/IPFIX data using Python pandas for statistical analysis, z-score anomaly detection, and hourly/daily traffic pattern profiling
detecting-evasion-techniques-in-endpoint-logs
Detects defense evasion techniques used by adversaries in endpoint logs including log tampering, timestomping, process injection, and security tool disabling. Use when investigating suspicious endpoint behavior, building detection rules for evasion tactics, or conducting threat hunting for stealthy adversary activity. Activates for requests involving evasion detection, defense evasion analysis, log tampering detection, or MITRE ATT&CK TA0005.
implementing-microsegmentation-with-guardicore
Implementing microsegmentation using Akamai Guardicore Segmentation to map application dependencies, create granular network policies, visualize east-west traffic flows, and enforce least-privilege communication between workloads across data centers and cloud.
ramdisk-tmpfs-patterns
暂无简介