redis-caching-and-pubsub
Architecting high-performance caching strategies, Cache-Aside/Write-Through/Write-Behind patterns, Redis Pub/Sub, Streams, eviction policies, distributed locking (Redlock), and memory optimization. Use when implementing Redis caching, real-time messaging, rate limiting, or session management.
- License 许可
- MIT
- 最近更新
- 2026-10-17
- Stars
- ★ 8
- Forks
- 1
01 前言 Front Matter
- name
- redis-caching-and-pubsub
- description
- Architecting high-performance caching strategies, Cache-Aside/Write-Through/Write-Behind patterns, Redis Pub/Sub, Streams, eviction policies, distributed locking (Redlock), and memory optimization. Use when implementing Redis caching, real-time messaging, rate limiting, or session management.
03 SKILL.md 正文
Redis Caching & Pub/Sub Architecture Guide
Production-grade guide for designing, implementing, and scaling Redis as a high-performance cache, session store, real-time pub/sub message broker, and distributed coordinator.
1. Caching Strategies & Design Patterns
1.1 Cache-Aside (Lazy Loading)
The application reads from the cache first. On a cache miss, it reads from the primary database, updates the cache, and returns the result.
+-------------+ 1. Read Key +-------+
| Application | --------------------> | Cache |
| | <-------------------- | |
+-------------+ 2. Cache Hit +-------+
|
| 3. Cache Miss
v
+---------------+
| Primary DB |
+---------------+
- Best For: Read-heavy workloads with unpredictable access patterns.
- Trade-off: Stale data risk if DB writes don't invalidate/update cache; higher latency on cache miss.
1.2 Write-Through Caching
The application writes data directly to the cache, and the cache synchronously writes to the database.
- Best For: Mission-critical reads where cache staleness cannot be tolerated.
- Trade-off: Higher write latency; potential cache pollution with infrequently read data.
1.3 Write-Behind (Write-Back) Caching
The application writes to the cache immediately, and an asynchronous process flushes cache updates to the underlying DB in batches.
- Best For: Write-heavy systems (e.g., analytics, metrics aggregators, counter updates).
- Trade-off: Risk of data loss if Redis crashes before flushing to primary DB.
1.4 Cache Stampede (Thundering Herd) Mitigation
When a popular cache key expires, thousands of concurrent requests hit the backend database simultaneously.
Solution A: Distributed Mutex / Lock
Acquire a lock to recompute the cache value; force other concurrent requests to wait or return stale data.
Solution B: Probabilistic Early Expiration (XFetch Algorithm)
As a key nears expiration, probabilistically refresh it in the background before it expires.
$$\Delta - \beta \cdot \ln(\text{rand}()) > \text{TTL}$$
2. Redis Data Structures & Scalable Usage Patterns
| Data Structure | Primary Use Cases | Recommended Max Size / Rule |
|---|---|---|
| Strings | Simple key-value, HTML fragments, JSON payloads, counters (INCRBY) |
Limit single string value to $< 100\text{ KB}$ |
| Hashes | User objects, entity fields, session attributes | Use fields instead of JSON strings for partial updates |
| Sets | Unique tags, IP blacklists, mutual friends, deduplication | Max set size $< 10,000$ members for fast bulk operations |
| Sorted Sets (ZSET) | Leaderboards, rate limit sliding windows, priority queues | High efficiency ($O(\log N)$ inserts/searches) |
| HyperLogLog | Unique visitor counting (Cardinality estimation) | Constant $12\text{ KB}$ footprint with $0.81%$ standard error |
| Bitmaps | User daily active status, feature toggles | $512\text{ MB}$ max space = $4.29$ billion flags |
| Streams | Append-only event logs, persistent pub/sub, message processing queues | Use XADD with explicit trimming (MAXLEN ~ 10000) |
3. Real-Time Messaging: Pub/Sub vs. Streams
Pub/Sub (At-Most-Once)
- Behavior: Fire-and-forget message delivery. Messages are not persisted. If a subscriber is disconnected during publication, the message is lost.
- Use Cases: Live UI updates, websocket notifications, ephemeral chat messages.
Redis Streams (At-Least-Once)
- Behavior: Append-only log with Consumer Groups, message ACK, pending entry lists (PEL), and replay capability.
- Use Cases: Async task processing, distributed event sourcing, reliable messaging workflows.
// Redis Streams Consumer Group Pattern (TypeScript Example)
import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS_URL!);
const STREAM_KEY = 'events:orders';
const GROUP_NAME = 'order-processing-group';
const CONSUMER_NAME = `worker-${process.pid}`;
async function initializeStreamGroup() {
try {
await redis.xgroup('CREATE', STREAM_KEY, GROUP_NAME, '$', 'MKSTREAM');
} catch (err: any) {
if (!err.message.includes('BUSYGROUP')) throw err;
}
}
async function processOrderEvents() {
await initializeStreamGroup();
while (true) {
// Read up to 10 new unconsumed messages for this group
const results = await redis.xreadgroup(
'GROUP', GROUP_NAME, CONSUMER_NAME,
'BLOCK', 2000,
'COUNT', 10,
'STREAMS', STREAM_KEY, '>'
) as any;
if (!results) continue;
for (const [stream, messages] of results) {
for (const [id, fields] of messages) {
try {
const payload = JSON.parse(fields[1]);
await handleOrderEvent(payload);
// Acknowledge successful processing
await redis.xack(STREAM_KEY, GROUP_NAME, id);
} catch (error) {
console.error(`Failed to process message ${id}:`, error);
// Failed messages remain in PEL for retry/dead-letter inspection
}
}
}
}
}
4. Distributed Locking (Redlock & Atomic Lua)
Single Instance Atomic Locking with Renewal & Release
To safely acquire and release locks, use a cryptographically random token and release via Lua script to avoid releasing another client's lock.
import Redis from 'ioredis';
import { randomBytes } from 'crypto';
export class RedisDistributedLock {
private redis: Redis;
constructor(redisClient: Redis) {
this.redis = redisClient;
}
/**
* Acquire a distributed lock.
*/
async acquire(key: string, ttlMs: number): Promise<string | null> {
const lockToken = randomBytes(16).toString('hex');
const acquired = await this.redis.set(
`lock:${key}`,
lockToken,
'PX',
ttlMs,
'NX'
);
return acquired === 'OK' ? lockToken : null;
}
/**
* Release lock safely using an atomic Lua script.
*/
async release(key: string, lockToken: string): Promise<boolean> {
const luaScript = `
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
`;
const result = await this.redis.eval(luaScript, 1, `lock:${key}`, lockToken);
return result === 1;
}
}
5. Eviction Policies & Memory Tuning
Redis Eviction Policies (maxmemory-policy)
volatile-lru: Evict keys with an explicit expiration using Least Recently Used algorithm.allkeys-lru: Recommended for general caching. Evicts any key based on LRU regardless of TTL.volatile-lfu/allkeys-lfu: Evicts based on Least Frequently Used frequency counters.noeviction: Returns Out-Of-Memory errors on writes when limit is reached. Required when Redis acts as a primary database.
Memory Optimization Rules
- Set
maxmemoryexplicitly (e.g., $75%$ of total instance RAM) to leave overhead for replication buffers and BGSAVE copy-on-write fork allocation. - Enable Listpack / Ziplist encoding for small Hashes and Sets:
hash-max-listpack-entries 512 hash-max-listpack-value 64
6. Anti-Patterns & Production Pitfalls
- ❌ Running
KEYS *in Production: Blocks single-threaded Redis execution loop. UseSCANwith cursor iteration instead. - ❌ Hot Key Bottlenecks: Single key accessed by thousands of app instances concurrently causing network NIC saturation. Fix: Implement local in-memory L1 cache (e.g., LRU cache) alongside Redis L2.
- ❌ Big Keys ($> 10\text{ MB}$ collections/strings): Unlinking/deleting large collections causes latency spikes. Use
UNLINKinstead ofDELfor async garbage collection. - ❌ Missing Key Namespaces: Keys without structured prefixes leads to collisions and unsafe bulk flush actions. Pattern:
app:domain:entity:id(e.g.,shop:user:profile:10948). - ❌ Unbounded Collection Growth: Sets, Lists, or Hashes left to grow indefinitely without TTLs or trimming logic.
04 同仓库更多 Skills
security-audit
Comprehensive security guidance, source code audit, vulnerability assessment, and penetration test review for codebases, APIs, microservices, CLI tools, libraries, and LLM-backed applications. Triggers on requests to audit a codebase, conduct security reviews, find vulnerabilities or flaws, pen-test source code, or when the user asks in Arabic (فحص أمني، تدقيق الكود أمنياً، مراجعة الثغرات، فحص أمان المشروع). Supports Guidance mode (focused review) and Full Audit mode (deterministic 6-phase harness).
malware-analysis-reversing
Production malware analysis workflows, reverse engineering methodologies, isolated sandbox triage, static/dynamic binary analysis, memory forensics, and YARA detection rule authoring.
application-security-pentesting
Production Application Security (AppSec) standards, OWASP Top 10 vulnerability remediation, SAST/DAST pipeline integration, secure code review, threat modeling, and defensive security posture.
oauth2-oidc-security-flows
Implement enterprise OAuth 2.1 and OpenID Connect (OIDC) authorization code flow with Proof Key for Code Exchange (PKCE), state and nonce verification, refresh token rotation, token revocation, and centralized IdP federation (Keycloak, Auth0, Okta). Trigger when building secure authentication, single sign-on (SSO), or third-party delegated authorization.
jwt-session-management-hardening
Harden JWT-based authentication and session lifecycles using asymmetric signing (RS256/EdDSA), JWKS rotation, strict claims validation, Redis-backed sliding window sessions, revocation lists, and secure HttpOnly cookie storage. Trigger when implementing token validation, session revocation, or hardening API authentication.
soc2-compliance-automation
Best practices for implementing SOC 2 Type II compliance controls, continuous evidence collection, infrastructure policy-as-code, and automated security monitoring. Use when automating Trust Services Criteria (Security, Availability, Confidentiality, Processing Integrity, Privacy), audit logging, IAM least privilege, or CI/CD security gates.