[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M792E3SRACS7PF35DGS6XTSF":3,"skill-same:01M792E3SRACS7PF35DGS6XTSF":61},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":38,"commitSha":39,"skillDirectory":40,"forksCount":41,"repositoryHtmlUrl":42,"repositoryHomepageUrl":6,"ownerHtmlUrl":43,"ownerBlogUrl":44,"ownerDescription":45,"updatedAt":46,"install":47,"resources":60},"01M792E3SRACS7PF35DGS6XTSF","langchain-performance-tuning",null,"Tune LangChain 1.0 \u002F LangGraph 1.0 Python chains and agents for throughput,\nlatency, and cost — streaming modes, explicit batch concurrency, semantic\nplus exact caches, persistent message history, and async-safe retriever\npatterns. Use when p95 latency exceeds target, batching \"does not work\",\ncost grows linearly with traffic, or a process restart wipes chat history.\nTrigger with \"langchain performance\", \"langchain slow batch\",\n\"langchain throughput\", \"langchain p95 latency\", \"semantic cache hit rate\".","jukeyman\u002Fjukeyman-skills","jukeyman","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F142960420?v=4","01M78FR4Y3DHMMCBCC0GXXNB3F","RICK JEFFERSON","01M78FR4Z8RVQRAD31Y5J7V6EB",1,"MIT","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fsaas-packs--langchain-py-pack--langchain-performance-tuning\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: langchain-performance-tuning\ndescription: |\n  Tune LangChain 1.0 \u002F LangGraph 1.0 Python chains and agents for throughput,\n  latency, and cost — streaming modes, explicit batch concurrency, semantic\n  plus exact caches, persistent message history, and async-safe retriever\n  patterns. Use when p95 latency exceeds target, batching \"does not work\",\n  cost grows linearly with traffic, or a process restart wipes chat history.\n  Trigger with \"langchain performance\", \"langchain slow batch\",\n  \"langchain throughput\", \"langchain p95 latency\", \"semantic cache hit rate\".\nallowed-tools: Read, Write, Edit, Bash(python:*), Bash(redis-cli:*)\nversion: 2.0.0\nlicense: MIT\nauthor: Jeremy Longshore \u003Cjeremy@intentsolutions.io>\ncompatible-with: claude-code, codex\ntags: [saas, langchain, langgraph, python, langchain-1.0, performance, caching, async]\n---\n\n# LangChain Performance Tuning\n\n## Overview\n\nAn engineer calls `chain.batch(inputs_1000)` expecting 1000 parallel LLM calls. Actual behavior: `Runnable.batch` and `Runnable.abatch` in LangChain 1.0 default to `max_concurrency=1`, so the 1000 inputs run **sequentially with bookkeeping overhead** — sometimes slower than a plain `for` loop. This is pain-catalog entry P08. The fix is one line:\n\n```python\n# Before: serial, ~1000 * per_call_latency\nawait chain.abatch(inputs)\n\n# After: 10x throughput at 10 providers' worth of concurrency\nawait chain.abatch(inputs, config={\"max_concurrency\": 10})\n```\n\nOther silent regressions in the same pain catalog: P48 (`invoke` inside `async def` blocks the FastAPI event loop), P22 (`InMemoryChatMessageHistory` loses every user's chat on restart), P62 (`RedisSemanticCache` at the default `score_threshold=0.95` returns under 5% hit rate), P59 (async retrievers leak connections on cancellation), P60 (`BackgroundTasks` fires *after* the response — wrong for per-token SSE), P01 (streaming token counts are only reliable on the `on_chat_model_end` event).\n\nThis skill wires a production performance baseline: explicit batch concurrency, async-only code paths, Redis-backed caches tuned on a golden set, persistent chat history with TTL, and TTFT instrumentation from `astream_events(version=\"v2\")`.\n\n## Prerequisites\n\n- Python 3.11+ with `langchain>=1.0,\u003C2`, `langgraph>=1.0,\u003C2`, `langchain-openai` or `langchain-anthropic`, `langchain-community`, `langchain-redis` or `redis>=5`.\n- A working LangChain 1.0 chain or LangGraph 1.0 graph that already passes functional tests.\n- Redis 7+ reachable from the app for cache and history (local Docker is fine for dev).\n- A FastAPI \u002F Starlette async endpoint, or an equivalent async entrypoint.\n- Observability: a place to emit metrics (Prometheus, OpenTelemetry, or LangSmith) — needed to measure TTFT, p95, and cache hit rate.\n\n## Instructions\n\n1. **Establish a latency budget and baseline.** Pick explicit targets before changing code: TTFT under 1s, p95 total under 5s, throughput over 20 req\u002Fs per worker, cost under $X per 1k interactions. Run a 5-minute load test with `locust` or `wrk` against the current chain and record p50 \u002F p95 \u002F p99 \u002F TTFT \u002F total cost. Without these numbers every downstream change is theater.\n\n2. **Convert every hot path to async (P48).** Inside `async def` handlers, replace `invoke`, `stream`, `batch`, `get_relevant_documents`, and `tool.run` with `ainvoke`, `astream` \u002F `astream_events(version=\"v2\")`, `abatch`, `aget_relevant_documents`, and `tool.arun`. See `references\u002Fasync-safety-checklist.md` for a grep pattern and a CI linter. Target: zero sync LangChain calls inside any async function.\n\n3. **Fix `.abatch()` concurrency (P08).** Every `.abatch` \u002F `.batch` call must pass `config={\"max_concurrency\": N}` where N is chosen from the provider table in `references\u002Fbatch-concurrency-per-provider.md` (Anthropic 10-20, OpenAI 20-50, local vLLM 100+). For multi-worker deploys, cap account-wide calls with a LiteLLM \u002F Portkey proxy or a Redis semaphore — `max_concurrency` only governs one process.\n\n4. **Instrument TTFT with `astream_events(version=\"v2\")` (P01).** Measure time to first token separately from total latency — user-perceived performance hinges on TTFT. Read usage metadata only on the `on_chat_model_end` event; per-chunk usage fields lag and are not reliable mid-stream.\n\n   ```python\n   from time import perf_counter\n   async def run(chain, query: str):\n       t0 = perf_counter(); ttft = None; tokens = 0\n       async for ev in chain.astream_events({\"input\": query}, version=\"v2\"):\n           if ev[\"event\"] == \"on_chat_model_stream\" and ttft is None:\n               ttft = perf_counter() - t0\n           if ev[\"event\"] == \"on_chat_model_end\":\n               tokens = ev[\"data\"][\"output\"].usage_metadata[\"total_tokens\"]\n       return {\"ttft_s\": ttft, \"total_s\": perf_counter() - t0, \"tokens\": tokens}\n   ```\n\n5. **Enable an exact LLM cache.** For deterministic (temperature=0) prompts, set `RedisCache` or `SQLiteCache` globally. LangChain 1.0 keys include the bound tools signature (P61 fix), which prevents cache poisoning when an agent's tool list changes. Always set an explicit TTL on Redis keys — default Redis keys are immortal.\n\n   ```python\n   from langchain_core.globals import set_llm_cache\n   from langchain_community.cache import RedisCache\n   import redis\n   set_llm_cache(RedisCache(redis.Redis.from_url(\"redis:\u002F\u002Fcache:6379\u002F0\")))\n   ```\n\n6. **Add a semantic cache with a tuned threshold (P62).** The `RedisSemanticCache` default `score_threshold=0.95` produces \u003C 5% hit rate on real traffic. Collect a 200-500 prompt golden set with labeled near-duplicates, measure cosine similarity with your embedding model, and pick the F1-maximizing threshold — typically **0.85-0.90** for `text-embedding-3-small`. Full procedure in `references\u002Fcache-tuning.md`. Do not run semantic cache behind `temperature > 0`; users will see prior random draws.\n\n7. **Replace `InMemoryChatMessageHistory` (P22).** Every production chat path must use `RedisChatMessageHistory` (with `ttl`) or a LangGraph checkpointer (`AsyncPostgresSaver` \u002F `AsyncSqliteSaver`). Add a restart test: mid-conversation, kill and restart the worker, assert the next user turn still sees prior messages. See `references\u002Fpersistent-history.md` for migration steps and trim policies.\n\n8. **Close retriever connection pools in FastAPI `lifespan` (P59).** Build the vector store once at startup, expose it via `app.state`, close it in the `finally` block. Never construct a retriever per request — cancellations leak pg connections.\n\n9. **Stream tokens with SSE, not `BackgroundTasks` (P60).** `BackgroundTasks` runs after the response body is flushed; per-token dispatch via it delivers tokens the client will never read. Use `EventSourceResponse` (sse-starlette) or a WebSocket and pipe events from `astream_events`.\n\n10. **Re-run the load test and diff the four metrics.** TTFT, p95, throughput, cost per 1k. If any regressed, revert that step and investigate — do not stack changes without verification. Execute in this order to isolate effects:\n\n    1. Run the baseline load test and save results.\n    2. Set `max_concurrency` on every `.abatch` call and re-run.\n    3. Add exact cache, re-run, check cache hit rate.\n    4. Configure semantic cache with tuned threshold, re-run, check hit rate again.\n    5. Verify persistent history survives a worker restart.\n\n### Throughput Tuning Table (starting values)\n\n| Provider | Safe `max_concurrency` | Ceiling signal |\n|----------|------------------------|-----------------|\n| Anthropic (sonnet-4.5\u002F4.6) | 10-20 | 429 `rate_limit_error` |\n| OpenAI (gpt-4o \u002F 4o-mini) | 20-50 | 429 + TPM exhaustion header |\n| OpenAI o1 \u002F reasoning | 2-5 | Cost + latency, not rate |\n| Google Gemini 1.5\u002F2.5 | 10-30 | 429 |\n| Cohere | 20-40 | 429 |\n| Local vLLM \u002F TGI | 100-500 (batch N≈32-64) | GPU KV-cache OOM |\n| Ollama on consumer GPU | 1-4 | Process queue backpressure |\n\n### Latency Breakdown Template\n\nRecord these for every change, not just total:\n\n| Metric | Target | Source |\n|--------|--------|--------|\n| TTFT p50 \u002F p95 | 500ms \u002F 1s | first `on_chat_model_stream` event |\n| Total p50 \u002F p95 | 2s \u002F 5s | end-to-end handler |\n| Tool-call p95 | \u003C 1s per tool | `on_tool_end` - `on_tool_start` |\n| Retriever p95 | \u003C 300ms | `on_retriever_end` - `on_retriever_start` |\n| Provider p95 | measure per model | split by LLM node |\n\n### Batch Sweet-Spot Numbers\n\n- Anthropic tier 2 chat: `max_concurrency=10` saturates at roughly 8 req\u002Fs, p95 doubles past 20.\n- OpenAI `gpt-4o-mini` tier 3: knee of the curve around `max_concurrency=30-40`; ~40 req\u002Fs throughput.\n- Local vLLM A100: server-side batch sweet spot `N=32-64`, client `max_concurrency=100+`.\n\nVerify on your own account — these are starting points, not promises.\n\n## Output\n\nDeliverables from running this skill end-to-end:\n\n- A `perf\u002F` directory with `baseline.json` and `tuned.json` load-test results.\n- All async handlers use `ainvoke` \u002F `astream_events` \u002F `abatch` with explicit `max_concurrency`.\n- `set_llm_cache` wired to `RedisCache` (exact) and optionally `RedisSemanticCache` (tuned threshold).\n- `RunnableWithMessageHistory` or LangGraph checkpointer backed by Redis or Postgres, with TTL.\n- FastAPI `lifespan` closing vector store pools on shutdown.\n- SSE endpoint streaming from `astream_events(version=\"v2\")`.\n- A `tests\u002Ftest_no_sync_in_async.py` CI guard (see async-safety reference).\n- Metrics exported: `ttft_seconds`, `total_latency_seconds`, `cache_hit_total`, `cache_miss_total`, `batch_concurrency_current`.\n- Runbook entry with the tuned `max_concurrency` per provider and the semantic-cache threshold, versioned in git.\n\n## Error Handling\n\n| Symptom | Root cause | Fix |\n|---------|-----------|-----|\n| `.abatch(inputs)` no faster than a `for` loop | `max_concurrency=1` default (P08) | Pass `config={\"max_concurrency\": N}` |\n| FastAPI TTFT collapses under load | Sync `invoke` inside `async def` (P48) | Switch to `ainvoke` \u002F `astream_events` |\n| Chat forgets prior turns after deploy | `InMemoryChatMessageHistory` (P22) | Move to `RedisChatMessageHistory` with TTL |\n| Semantic cache hit rate \u003C 5% | `score_threshold=0.95` default (P62) | Tune on golden set to 0.85-0.90 |\n| pg pool exhausted hours into load test | Retriever not closed on cancel (P59) | Close vector store in FastAPI `lifespan` |\n| SSE client sees zero tokens | Dispatching via `BackgroundTasks` (P60) | Use `EventSourceResponse` and `astream_events` |\n| Per-chunk token counts fluctuate | Usage metadata lags during stream (P01) | Read only on `on_chat_model_end` |\n| 429 storm after tuning concurrency | Per-worker limit * N workers > account RPM | Add LiteLLM\u002FPortkey proxy or Redis semaphore |\n| Semantic cache returns off-brand output | Cache hit on `temperature > 0` route | Disable semantic cache or force temperature=0 |\n| Cache poisoning after tool change | Missing tools in cache key | Upgrade LangChain to 1.0.x post-P61 fix |\n\n## Examples\n\n**Example 1 — Fix a sequential batch job.**\n\n```python\n# Before — 1000 items, 18 minutes end-to-end\nresults = await chain.abatch(inputs)\n\n# After — 1000 items, ~2 minutes; Anthropic tier-2 account, N=10\nresults = await chain.abatch(inputs, config={\"max_concurrency\": 10})\n```\n\n**Example 2 — Wire persistent history and an exact cache on a FastAPI app.**\n\n```python\nfrom contextlib import asynccontextmanager\nfrom fastapi import FastAPI\nfrom langchain_core.globals import set_llm_cache\nfrom langchain_core.runnables.history import RunnableWithMessageHistory\nfrom langchain_community.cache import RedisCache\nfrom langchain_community.chat_message_histories import RedisChatMessageHistory\nimport redis\n\n@asynccontextmanager\nasync def lifespan(app: FastAPI):\n    r = redis.Redis.from_url(\"redis:\u002F\u002Fcache:6379\u002F0\")\n    set_llm_cache(RedisCache(r))\n    app.state.r = r\n    yield\n    r.close()\n\napp = FastAPI(lifespan=lifespan)\n\ndef history_for(session_id: str) -> RedisChatMessageHistory:\n    return RedisChatMessageHistory(\n        session_id=session_id,\n        url=\"redis:\u002F\u002Fhistory:6379\u002F2\",\n        ttl=60 * 60 * 24 * 14,\n    )\n\nchain_with_history = RunnableWithMessageHistory(\n    base_chain, history_for,\n    input_messages_key=\"input\",\n    history_messages_key=\"history\",\n)\n```\n\n**Example 3 — Stream tokens with measured TTFT.**\n\n```python\nfrom sse_starlette.sse import EventSourceResponse\nfrom time import perf_counter\n\n@app.post(\"\u002Fchat\")\nasync def chat(req: ChatReq):\n    async def gen():\n        t0 = perf_counter()\n        async for ev in chain_with_history.astream_events(\n            {\"input\": req.text},\n            config={\"configurable\": {\"session_id\": req.session_id}},\n            version=\"v2\",\n        ):\n            if ev[\"event\"] == \"on_chat_model_stream\":\n                yield {\"data\": ev[\"data\"][\"chunk\"].content}\n        app.state.r.incrbyfloat(\"ttft_sum_s\", perf_counter() - t0)\n    return EventSourceResponse(gen())\n```\n\n## Resources\n\n- [One-pager](references\u002Fone-pager.md) — problem \u002F solution \u002F key features snapshot.\n- [batch-concurrency-per-provider](references\u002Fbatch-concurrency-per-provider.md) — per-provider `max_concurrency` table, sweep procedure, semaphore patterns.\n- [cache-tuning](references\u002Fcache-tuning.md) — exact vs semantic, Redis key design, golden-set threshold procedure, TTL strategy.\n- [persistent-history](references\u002Fpersistent-history.md) — Redis \u002F Postgres \u002F LangGraph checkpointer migration off `InMemoryChatMessageHistory`.\n- [async-safety-checklist](references\u002Fasync-safety-checklist.md) — sync-in-async grep + linter, lifespan pool cleanup, SSE vs `BackgroundTasks`.\n- [LangChain streaming \u002F batching](https:\u002F\u002Fpython.langchain.com\u002Fdocs\u002Fhow_to\u002Fstreaming\u002F#batching) — official docs for `Runnable.batch` and streaming modes.\n- [LangChain caching](https:\u002F\u002Fpython.langchain.com\u002Fdocs\u002Fhow_to\u002Fllm_caching\u002F) — `set_llm_cache`, Redis and SQLite backends.\n- [LangGraph checkpointers](https:\u002F\u002Flangchain-ai.github.io\u002Flanggraph\u002Fhow-tos\u002Fpersistence\u002F) — persistence for graph state.\n- Companion skills in `langchain-py-pack`: `langchain-model-inference` (token accounting), `langchain-embeddings-search` (retrieval tuning), `langchain-middleware-patterns` (tool-signature cache keying, P61).\n","# LangChain Performance Tuning\n\n## Overview\n\nAn engineer calls `chain.batch(inputs_1000)` expecting 1000 parallel LLM calls. Actual behavior: `Runnable.batch` and `Runnable.abatch` in LangChain 1.0 default to `max_concurrency=1`, so the 1000 inputs run **sequentially with bookkeeping overhead** — sometimes slower than a plain `for` loop. This is pain-catalog entry P08. The fix is one line:\n\n```python\n# Before: serial, ~1000 * per_call_latency\nawait chain.abatch(inputs)\n\n# After: 10x throughput at 10 providers' worth of concurrency\nawait chain.abatch(inputs, config={\"max_concurrency\": 10})\n```\n\nOther silent regressions in the same pain catalog: P48 (`invoke` inside `async def` blocks the FastAPI event loop), P22 (`InMemoryChatMessageHistory` loses every user's chat on restart), P62 (`RedisSemanticCache` at the default `score_threshold=0.95` returns under 5% hit rate), P59 (async retrievers leak connections on cancellation), P60 (`BackgroundTasks` fires *after* the response — wrong for per-token SSE), P01 (streaming token counts are only reliable on the `on_chat_model_end` event).\n\nThis skill wires a production performance baseline: explicit batch concurrency, async-only code paths, Redis-backed caches tuned on a golden set, persistent chat history with TTL, and TTFT instrumentation from `astream_events(version=\"v2\")`.\n\n## Prerequisites\n\n- Python 3.11+ with `langchain>=1.0,\u003C2`, `langgraph>=1.0,\u003C2`, `langchain-openai` or `langchain-anthropic`, `langchain-community`, `langchain-redis` or `redis>=5`.\n- A working LangChain 1.0 chain or LangGraph 1.0 graph that already passes functional tests.\n- Redis 7+ reachable from the app for cache and history (local Docker is fine for dev).\n- A FastAPI \u002F Starlette async endpoint, or an equivalent async entrypoint.\n- Observability: a place to emit metrics (Prometheus, OpenTelemetry, or LangSmith) — needed to measure TTFT, p95, and cache hit rate.\n\n## Instructions\n\n1. **Establish a latency budget and baseline.** Pick explicit targets before changing code: TTFT under 1s, p95 total under 5s, throughput over 20 req\u002Fs per worker, cost under $X per 1k interactions. Run a 5-minute load test with `locust` or `wrk` against the current chain and record p50 \u002F p95 \u002F p99 \u002F TTFT \u002F total cost. Without these numbers every downstream change is theater.\n\n2. **Convert every hot path to async (P48).** Inside `async def` handlers, replace `invoke`, `stream`, `batch`, `get_relevant_documents`, and `tool.run` with `ainvoke`, `astream` \u002F `astream_events(version=\"v2\")`, `abatch`, `aget_relevant_documents`, and `tool.arun`. See `references\u002Fasync-safety-checklist.md` for a grep pattern and a CI linter. Target: zero sync LangChain calls inside any async function.\n\n3. **Fix `.abatch()` concurrency (P08).** Every `.abatch` \u002F `.batch` call must pass `config={\"max_concurrency\": N}` where N is chosen from the provider table in `references\u002Fbatch-concurrency-per-provider.md` (Anthropic 10-20, OpenAI 20-50, local vLLM 100+). For multi-worker deploys, cap account-wide calls with a LiteLLM \u002F Portkey proxy or a Redis semaphore — `max_concurrency` only governs one process.\n\n4. **Instrument TTFT with `astream_events(version=\"v2\")` (P01).** Measure time to first token separately from total latency — user-perceived performance hinges on TTFT. Read usage metadata only on the `on_chat_model_end` event; per-chunk usage fields lag and are not reliable mid-stream.\n\n   ```python\n   from time import perf_counter\n   async def run(chain, query: str):\n       t0 = perf_counter(); ttft = None; tokens = 0\n       async for ev in chain.astream_events({\"input\": query}, version=\"v2\"):\n           if ev[\"event\"] == \"on_chat_model_stream\" and ttft is None:\n               ttft = perf_counter() - t0\n           if ev[\"event\"] == \"on_chat_model_end\":\n               tokens = ev[\"data\"][\"output\"].usage_metadata[\"total_tokens\"]\n       return {\"ttft_s\": ttft, \"total_s\": perf_counter() - t0, \"tokens\": tokens}\n   ```\n\n5. **Enable an exact LLM cache.** For deterministic (temperature=0) prompts, set `RedisCache` or `SQLiteCache` globally. LangChain 1.0 keys include the bound tools signature (P61 fix), which prevents cache poisoning when an agent's tool list changes. Always set an explicit TTL on Redis keys — default Redis keys are immortal.\n\n   ```python\n   from langchain_core.globals import set_llm_cache\n   from langchain_community.cache import RedisCache\n   import redis\n   set_llm_cache(RedisCache(redis.Redis.from_url(\"redis:\u002F\u002Fcache:6379\u002F0\")))\n   ```\n\n6. **Add a semantic cache with a tuned threshold (P62).** The `RedisSemanticCache` default `score_threshold=0.95` produces \u003C 5% hit rate on real traffic. Collect a 200-500 prompt golden set with labeled near-duplicates, measure cosine similarity with your embedding model, and pick the F1-maximizing threshold — typically **0.85-0.90** for `text-embedding-3-small`. Full procedure in `references\u002Fcache-tuning.md`. Do not run semantic cache behind `temperature > 0`; users will see prior random draws.\n\n7. **Replace `InMemoryChatMessageHistory` (P22).** Every production chat path must use `RedisChatMessageHistory` (with `ttl`) or a LangGraph checkpointer (`AsyncPostgresSaver` \u002F `AsyncSqliteSaver`). Add a restart test: mid-conversation, kill and restart the worker, assert the next user turn still sees prior messages. See `references\u002Fpersistent-history.md` for migration steps and trim policies.\n\n8. **Close retriever connection pools in FastAPI `lifespan` (P59).** Build the vector store once at startup, expose it via `app.state`, close it in the `finally` block. Never construct a retriever per request — cancellations leak pg connections.\n\n9. **Stream tokens with SSE, not `BackgroundTasks` (P60).** `BackgroundTasks` runs after the response body is flushed; per-token dispatch via it delivers tokens the client will never read. Use `EventSourceResponse` (sse-starlette) or a WebSocket and pipe events from `astream_events`.\n\n10. **Re-run the load test and diff the four metrics.** TTFT, p95, throughput, cost per 1k. If any regressed, revert that step and investigate — do not stack changes without verification. Execute in this order to isolate effects:\n\n    1. Run the baseline load test and save results.\n    2. Set `max_concurrency` on every `.abatch` call and re-run.\n    3. Add exact cache, re-run, check cache hit rate.\n    4. Configure semantic cache with tuned threshold, re-run, check hit rate again.\n    5. Verify persistent history survives a worker restart.\n\n### Throughput Tuning Table (starting values)\n\n| Provider | Safe `max_concurrency` | Ceiling signal |\n|----------|------------------------|-----------------|\n| Anthropic (sonnet-4.5\u002F4.6) | 10-20 | 429 `rate_limit_error` |\n| OpenAI (gpt-4o \u002F 4o-mini) | 20-50 | 429 + TPM exhaustion header |\n| OpenAI o1 \u002F reasoning | 2-5 | Cost + latency, not rate |\n| Google Gemini 1.5\u002F2.5 | 10-30 | 429 |\n| Cohere | 20-40 | 429 |\n| Local vLLM \u002F TGI | 100-500 (batch N≈32-64) | GPU KV-cache OOM |\n| Ollama on consumer GPU | 1-4 | Process queue backpressure |\n\n### Latency Breakdown Template\n\nRecord these for every change, not just total:\n\n| Metric | Target | Source |\n|--------|--------|--------|\n| TTFT p50 \u002F p95 | 500ms \u002F 1s | first `on_chat_model_stream` event |\n| Total p50 \u002F p95 | 2s \u002F 5s | end-to-end handler |\n| Tool-call p95 | \u003C 1s per tool | `on_tool_end` - `on_tool_start` |\n| Retriever p95 | \u003C 300ms | `on_retriever_end` - `on_retriever_start` |\n| Provider p95 | measure per model | split by LLM node |\n\n### Batch Sweet-Spot Numbers\n\n- Anthropic tier 2 chat: `max_concurrency=10` saturates at roughly 8 req\u002Fs, p95 doubles past 20.\n- OpenAI `gpt-4o-mini` tier 3: knee of the curve around `max_concurrency=30-40`; ~40 req\u002Fs throughput.\n- Local vLLM A100: server-side batch sweet spot `N=32-64`, client `max_concurrency=100+`.\n\nVerify on your own account — these are starting points, not promises.\n\n## Output\n\nDeliverables from running this skill end-to-end:\n\n- A `perf\u002F` directory with `baseline.json` and `tuned.json` load-test results.\n- All async handlers use `ainvoke` \u002F `astream_events` \u002F `abatch` with explicit `max_concurrency`.\n- `set_llm_cache` wired to `RedisCache` (exact) and optionally `RedisSemanticCache` (tuned threshold).\n- `RunnableWithMessageHistory` or LangGraph checkpointer backed by Redis or Postgres, with TTL.\n- FastAPI `lifespan` closing vector store pools on shutdown.\n- SSE endpoint streaming from `astream_events(version=\"v2\")`.\n- A `tests\u002Ftest_no_sync_in_async.py` CI guard (see async-safety reference).\n- Metrics exported: `ttft_seconds`, `total_latency_seconds`, `cache_hit_total`, `cache_miss_total`, `batch_concurrency_current`.\n- Runbook entry with the tuned `max_concurrency` per provider and the semantic-cache threshold, versioned in git.\n\n## Error Handling\n\n| Symptom | Root cause | Fix |\n|---------|-----------|-----|\n| `.abatch(inputs)` no faster than a `for` loop | `max_concurrency=1` default (P08) | Pass `config={\"max_concurrency\": N}` |\n| FastAPI TTFT collapses under load | Sync `invoke` inside `async def` (P48) | Switch to `ainvoke` \u002F `astream_events` |\n| Chat forgets prior turns after deploy | `InMemoryChatMessageHistory` (P22) | Move to `RedisChatMessageHistory` with TTL |\n| Semantic cache hit rate \u003C 5% | `score_threshold=0.95` default (P62) | Tune on golden set to 0.85-0.90 |\n| pg pool exhausted hours into load test | Retriever not closed on cancel (P59) | Close vector store in FastAPI `lifespan` |\n| SSE client sees zero tokens | Dispatching via `BackgroundTasks` (P60) | Use `EventSourceResponse` and `astream_events` |\n| Per-chunk token counts fluctuate | Usage metadata lags during stream (P01) | Read only on `on_chat_model_end` |\n| 429 storm after tuning concurrency | Per-worker limit * N workers > account RPM | Add LiteLLM\u002FPortkey proxy or Redis semaphore |\n| Semantic cache returns off-brand output | Cache hit on `temperature > 0` route | Disable semantic cache or force temperature=0 |\n| Cache poisoning after tool change | Missing tools in cache key | Upgrade LangChain to 1.0.x post-P61 fix |\n\n## Examples\n\n**Example 1 — Fix a sequential batch job.**\n\n```python\n# Before — 1000 items, 18 minutes end-to-end\nresults = await chain.abatch(inputs)\n\n# After — 1000 items, ~2 minutes; Anthropic tier-2 account, N=10\nresults = await chain.abatch(inputs, config={\"max_concurrency\": 10})\n```\n\n**Example 2 — Wire persistent history and an exact cache on a FastAPI app.**\n\n```python\nfrom contextlib import asynccontextmanager\nfrom fastapi import FastAPI\nfrom langchain_core.globals import set_llm_cache\nfrom langchain_core.runnables.history import RunnableWithMessageHistory\nfrom langchain_community.cache import RedisCache\nfrom langchain_community.chat_message_histories import RedisChatMessageHistory\nimport redis\n\n@asynccontextmanager\nasync def lifespan(app: FastAPI):\n    r = redis.Redis.from_url(\"redis:\u002F\u002Fcache:6379\u002F0\")\n    set_llm_cache(RedisCache(r))\n    app.state.r = r\n    yield\n    r.close()\n\napp = FastAPI(lifespan=lifespan)\n\ndef history_for(session_id: str) -> RedisChatMessageHistory:\n    return RedisChatMessageHistory(\n        session_id=session_id,\n        url=\"redis:\u002F\u002Fhistory:6379\u002F2\",\n        ttl=60 * 60 * 24 * 14,\n    )\n\nchain_with_history = RunnableWithMessageHistory(\n    base_chain, history_for,\n    input_messages_key=\"input\",\n    history_messages_key=\"history\",\n)\n```\n\n**Example 3 — Stream tokens with measured TTFT.**\n\n```python\nfrom sse_starlette.sse import EventSourceResponse\nfrom time import perf_counter\n\n@app.post(\"\u002Fchat\")\nasync def chat(req: ChatReq):\n    async def gen():\n        t0 = perf_counter()\n        async for ev in chain_with_history.astream_events(\n            {\"input\": req.text},\n            config={\"configurable\": {\"session_id\": req.session_id}},\n            version=\"v2\",\n        ):\n            if ev[\"event\"] == \"on_chat_model_stream\":\n                yield {\"data\": ev[\"data\"][\"chunk\"].content}\n        app.state.r.incrbyfloat(\"ttft_sum_s\", perf_counter() - t0)\n    return EventSourceResponse(gen())\n```\n\n## Resources\n\n- [One-pager](references\u002Fone-pager.md) — problem \u002F solution \u002F key features snapshot.\n- [batch-concurrency-per-provider](references\u002Fbatch-concurrency-per-provider.md) — per-provider `max_concurrency` table, sweep procedure, semaphore patterns.\n- [cache-tuning](references\u002Fcache-tuning.md) — exact vs semantic, Redis key design, golden-set threshold procedure, TTL strategy.\n- [persistent-history](references\u002Fpersistent-history.md) — Redis \u002F Postgres \u002F LangGraph checkpointer migration off `InMemoryChatMessageHistory`.\n- [async-safety-checklist](references\u002Fasync-safety-checklist.md) — sync-in-async grep + linter, lifespan pool cleanup, SSE vs `BackgroundTasks`.\n- [LangChain streaming \u002F batching](https:\u002F\u002Fpython.langchain.com\u002Fdocs\u002Fhow_to\u002Fstreaming\u002F#batching) — official docs for `Runnable.batch` and streaming modes.\n- [LangChain caching](https:\u002F\u002Fpython.langchain.com\u002Fdocs\u002Fhow_to\u002Fllm_caching\u002F) — `set_llm_cache`, Redis and SQLite backends.\n- [LangGraph checkpointers](https:\u002F\u002Flangchain-ai.github.io\u002Flanggraph\u002Fhow-tos\u002Fpersistence\u002F) — persistence for graph state.\n- Companion skills in `langchain-py-pack`: `langchain-model-inference` (token accounting), `langchain-embeddings-search` (retrieval tuning), `langchain-middleware-patterns` (tool-signature cache keying, P61).\n",{"name":5,"tags":24,"author":33,"license":15,"version":34,"description":35,"allowed-tools":36,"compatible-with":37},[25,26,27,28,29,30,31,32],"saas","langchain","langgraph","python","langchain-1.0","performance","caching","async","Jeremy Longshore \u003Cjeremy@intentsolutions.io>","2.0.0","Tune LangChain 1.0 \u002F LangGraph 1.0 Python chains and agents for throughput,\nlatency, and cost — streaming modes, explicit batch concurrency, semantic\nplus exact caches, persistent message history, and async-safe retriever\npatterns. Use when p95 latency exceeds target, batching \"does not work\",\ncost grows linearly with traffic, or a process restart wipes chat history.\nTrigger with \"langchain performance\", \"langchain slow batch\",\n\"langchain throughput\", \"langchain p95 latency\", \"semantic cache hit rate\".\n","Read, Write, Edit, Bash(python:*), Bash(redis-cli:*)","claude-code, codex","main","99825ad3f3bcc6597ddcc7294a14e6e922f2b67b","skills\u002Fsaas-packs--langchain-py-pack--langchain-performance-tuning",0,"https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills","https:\u002F\u002Fgithub.com\u002Fjukeyman","https:\u002F\u002FRJBUSINESSSOLUTIONS.ORG","Credit and Finance Expert | Developer\r\n\r\nI'm Rick Jefferson, a seasoned credit and finance expert with over a decade of experience in financial literacy,","2026-07-13 07:03:29",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add jukeyman\u002Fjukeyman-skills --skill langchain-performance-tuning",{"tool":53,"command":54},"bunx","bunx skills add jukeyman\u002Fjukeyman-skills --skill langchain-performance-tuning",{"tool":56,"command":57},"pnpm","pnpm dlx skills add jukeyman\u002Fjukeyman-skills --skill langchain-performance-tuning","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fsaas-packs--langchain-py-pack--langchain-performance-tuning\u002FSKILL.md",true,[],[62,69,76,83,90,98],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M79ANS2SM329NEP5ZCX106RR","zustand-store-creator","Create zustand store creator operations. Auto-activating skill for Frontend Development.\nTriggers on: zustand store creator, zustand store creator\nPart of the Frontend Development skill category. Use when working with zustand store creator functionality. Trigger with phrases like \"zustand store creator\", \"zustand creator\", \"zustand\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzustand-store-creator\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M79AMZKP4YDEZ7EHG1Y7PP74","zero-trust-config-helper","Configure with zero trust config helper operations. Auto-activating skill for Security Advanced.\nTriggers on: zero trust config helper, zero trust config helper\nPart of the Security Advanced skill category. Use when configuring systems or services. Trigger with phrases like \"zero trust config helper\", \"zero helper\", \"zero\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzero-trust-config-helper\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M79AMJSBCEJQNBTYNAHYN5HM","zapier-integration-helper","Assist with zapier integration helper operations. Auto-activating skill for Business Automation.\nTriggers on: zapier integration helper, zapier integration helper\nPart of the Business Automation skill category. Use when working with APIs or building integrations. Trigger with phrases like \"zapier integration helper\", \"zapier helper\", \"zapier\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzapier-integration-helper\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M79AHJZGVMARS13VFQYRR1MP","yaml-config-validator","Validate yaml config validator operations. Auto-activating skill for DevOps Basics.\nTriggers on: yaml config validator, yaml config validator\nPart of the DevOps Basics skill category. Use when configuring systems or services. Trigger with phrases like \"yaml config validator\", \"yaml validator\", \"yaml\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fyaml-config-validator\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":94,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":19},"01M79AHGDZRN0020RDQSQDPAM8","xvary-stock-research","Thesis-driven equity analysis from public SEC EDGAR and market data; \u002Fanalyze, \u002Fscore, \u002Fcompare workflows with bundled Python tools (Claude Code, Cursor, Codex).","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxvary-stock-research\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxvary-stock-research\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":19},"01M79AHCYJ9FV9RMHQ24HDRN33","xss-vulnerability-scanner","Scan xss vulnerability scanner operations. Auto-activating skill for Security Fundamentals.\nTriggers on: xss vulnerability scanner, xss vulnerability scanner\nPart of the Security Fundamentals skill category. Use when working with xss vulnerability scanner functionality. Trigger with phrases like \"xss vulnerability scanner\", \"xss scanner\", \"xss\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxss-vulnerability-scanner\u002FSKILL.md",[],[]]