[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M784SDEW9G4REFT90HT4EDM2":3,"skill-same:01M784SDEW9G4REFT90HT4EDM2":58},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":37,"commitSha":38,"skillDirectory":39,"forksCount":40,"repositoryHtmlUrl":41,"repositoryHomepageUrl":6,"ownerHtmlUrl":42,"ownerBlogUrl":43,"ownerDescription":6,"updatedAt":44,"install":45,"resources":57},"01M784SDEW9G4REFT90HT4EDM2","fix-codex-litellm",null,"Use when the codex.plan.ai litellm plugin (Codex OAuth custom provider) is throwing errors — HTTP 400\u002F4xx\u002F5xx from the Codex \u002Fresponses endpoint, request\u002Fresponse translation failures, model-name or auth errors — to diagnose, fix the provider fork, and redeploy codex.plan.ai.","planailabs\u002Fnixos-infra","planailabs","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F261949590?v=4","01M784QMNK8YTQBDBX1G4QWPBZ","plan.ai Labs","01M784S1HQN2W6B9ZTXNA7N91T",1,"MIT","https:\u002F\u002Fgithub.com\u002Fplanailabs\u002Fnixos-infra\u002Fblob\u002Ftrunk\u002F.claude\u002Fskills\u002Ffix-codex-litellm\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: fix-codex-litellm\ndescription: Use when the codex.plan.ai litellm plugin (Codex OAuth custom provider) is throwing errors — HTTP 400\u002F4xx\u002F5xx from the Codex \u002Fresponses endpoint, request\u002Fresponse translation failures, model-name or auth errors — to diagnose, fix the provider fork, and redeploy codex.plan.ai.\nversion: 1.0.0\nauthor: plan.ai\nlicense: MIT\nmetadata:\n  hermes:\n    tags: [nixos, infrastructure, litellm, codex, deploy]\n    related_skills: [update-nixos]\nuser_invocable: true\n---\n\n# Fix codex.plan.ai litellm plugin\n\n## Overview\n\n`codex.plan.ai` runs a litellm proxy (official `ghcr.io\u002Fberriai\u002Flitellm-database`\nimage, DB mode) fronting ChatGPT's Codex backend through a **custom OAuth\nprovider**. The provider is a pinned fork —\n`mkg20001\u002Flitellm-codex-oauth-provider` — that bridges chat-completions calls to\nthe Codex `\u002Fresponses` API (`https:\u002F\u002Fchatgpt.com\u002Fbackend-api\u002Fcodex\u002Fresponses`).\n\nErrors surface in the calling app's logs (e.g. `openclaw` on other hosts) and in\nthe litellm container logs on `codex.plan.ai`. Most bugs live in the **fork's\ntranslation code**, not in this repo's Nix. This skill diagnoses the failure,\nfixes it at the right layer, and redeploys.\n\nRun all repo commands from the `nixos-infra` root. The git\u002Fgh account here is\n`mkg20001`, which has **ADMIN** on the fork — fixes go directly to the fork's\n`main`.\n\nFor a **check** rather than a repair — is the proxy serving right now, and which\nlayer is down — use the `codex-healthcheck` skill first; it probes\n`\u002Fv1\u002Fmodels` + one tiny chat completion and only sends you here for the deep\nlayer. Come back to this skill for the diagnosis, the fork fix and the redeploy.\n\n## Architecture map\n\n- `codex\u002Fdefault.nix` — the host. Builds the litellm config dir at each start\n  (`litellm-codex-config` oneshot → `genLitellmConfig`), runs the container, and\n  defines the `codex_handler.py` shim. Config dir lives at `\u002Fvar\u002Flib\u002Flitellm-codex`,\n  bind-mounted read-only into `\u002Fconfig`.\n- `pkgs\u002Flitellm-codex-oauth-provider.nix` — packages the fork (pinned `rev` +\n  `hash`). Its source is copied into the config dir (importable via\n  `PYTHONPATH=\u002Fconfig`).\n- Fork internals (clone to inspect — see Step 3):\n  - `prompts.py` — **request** side: chat-completions messages → Codex `input`.\n    `derive_instructions`, `_to_codex_input_items`, `_to_responses_content`.\n  - `adapter.py` — **response** side: Codex `\u002Fresponses` payload (JSON or SSE) →\n    litellm `ModelResponse`. `transform_response`, `convert_sse_to_json`.\n  - `models.py` \u002F `model_map.py` — live model discovery + slug mapping.\n  - `auth.py` — OAuth access-token refresh (rewrites `\u002Froot\u002F.codex\u002Fauth.json`).\n  - `streaming_utils.py`, `sse_utils.py` — SSE handling.\n  - `images.py` — vision: downloads remote `http(s)` image URLs and inlines\n    them as base64 `data:` URLs (the backend can't reliably fetch arbitrary\n    public URLs — the Codex CLI only ever sends data URLs).\n  - `provider.py` also implements `image_generation`\u002F`aimage_generation` on\n    top of the Responses-API hosted `image_generation` tool (there is no\n    dedicated images endpoint; the b64 image arrives in\n    `image_generation_call` output items' `result` field).\n- Handler shim (`codexHandlerPy` in `codex\u002Fdefault.nix`) — re-exports the\n  provider instance AND removes discovered model slugs from\n  `litellm.open_ai_chat_completion_models` so litellm's dispatch doesn't route\n  codex models to its built-in OpenAI handler. Also patches\n  `litellm.images.main.get_optional_params_image_gen`: for custom providers\n  litellm returns `{}` there (its provider-config registry is a hardcoded\n  if\u002Felif over built-ins), silently dropping standard image params\n  (`n`\u002F`quality`\u002F`size`\u002F`style`) before they reach the custom handler.\n\n## Step 1: Capture the error\n\nGet the exact error and its shape. The `param`\u002Ftraceback tells you which layer.\n\nAsk the user for the failing log line if not already provided, and pull the\ncontainer logs from the host:\n\n```bash\nssh root@codex.plan.ai 'journalctl -u podman-litellm --no-pager -n 200'\nssh root@codex.plan.ai 'journalctl -u litellm-codex-config --no-pager -n 80'\n```\n\nWhen you drive this from **reagent** (rather than an interactive terminal) on\nthis project: run shell tasks with `devshell: false` — the flake has no default\ndevshell and reagent's automatic `nix develop` wrapper fails there. Run\n`ssh-keyscan codex.plan.ai >> ~\u002F.ssh\u002Fknown_hosts` once (and for any other host\nyou deploy to) before the first ssh, or the host key prompt eats the command.\n\n`private\u002F` is a git submodule and `private\u002F*.nix` (e.g. `private\u002Fcodex.nix`) is\nread from it — run `git submodule update --init` on a fresh checkout, or the\nbuild fails (see Step 6).\n\n## Step 2: Classify the failure\n\n- **Request-translation (most common).** HTTP 400 from `...\u002Fcodex\u002Fresponses`\n  with `invalid_request_error` naming an `input[...]` param, e.g.\n  `Invalid value: 'text' ... Supported values are: 'input_text', ...` at\n  `input[N].content[0].type`. → fix `prompts.py`. (Known example: list-form\n  message `content` parts forwarded with chat-completions types instead of\n  Responses-API types `input_text`\u002F`input_image`\u002F`output_text`.)\n- **Response-translation.** Errors after a 200 from Codex — missing choices,\n  bad tool_calls, SSE parse failures, wrong usage. → fix `adapter.py`.\n- **Model not supported.** `model is not supported when using Codex with a\n  ChatGPT account`, or a slug that 400s. Codex slugs change over time and are\n  **discovered live** — nothing is hardcoded. Regenerate the config:\n  `ssh root@codex.plan.ai systemctl restart podman-litellm` (re-runs\n  `litellm-codex-config`). Check its journal for the discovered model_list.\n- **Missing\u002Fhidden models, or 404 \"Model not found\" on \u002Fresponses.** The\n  backend version-gates BOTH endpoints: `\u002Fcodex\u002Fmodels` by the `client_version`\n  query param and `\u002Fcodex\u002Fresponses` by the `version` header. genLitellmConfig\n  resolves the latest codex release tag from GitHub at each start and shares it\n  via `\u002Fvar\u002Flib\u002Flitellm-codex\u002Fclient-version.env` (fallback:\n  `pkgs.codex.version`, which lags upstream). If new models are missing, check\n  the journal's `codex client_version:` line first.\n- **Broken\u002Fempty pricing (`priced models:` empty in the config journal).**\n  Prices are scraped from the OpenAI pricing page by regex in\n  `codex\u002Fdefault.nix` (`fetch_pricing`); the page's encoded-row format has\n  changed more than once (column added, `\"-\"` placeholders for missing prices).\n  Fetch the page, inspect the encoded rows around a known slug, adjust the\n  parser, and test it standalone before deploying.\n- **Dispatch \u002F \"api_key must be set\".** Error names the custom provider\n  (`CodexException`) but the traceback is in `llms\u002Fopenai\u002Fopenai.py` — a codex\n  slug leaked into `litellm.open_ai_chat_completion_models`. → the shim in\n  `codex\u002Fdefault.nix` (`codexHandlerPy`) should strip it; verify\n  `available_model_slugs()` covers the slug.\n- **Auth \u002F token.** 401\u002Fexpired. The provider refreshes `\u002Froot\u002F.codex\u002Fauth.json`\n  in place (mounted rw). If refresh is broken, re-run `codex login` on the host.\n  Fixes to refresh logic go in `auth.py`.\n- **Vision \u002F images.** Three distinct shapes:\n  - *Client refuses to send images* (\"model does not support vision\"): litellm\n    only advertises what `model_info` declares. genLitellmConfig derives it per\n    model from the discovery payload (`supports_vision` ← `input_modalities`\n    contains `\"image\"`, `max_input_tokens` ← `context_window`); check\n    `\u002Fmodel\u002Finfo` on the proxy first.\n  - *Remote image URL 400s* with `\"Error while downloading file. Upstream\n    status code: NNN.\"`, param `url`: the backend tried (and failed) to fetch\n    the URL server-side. `images.py` should have inlined it as a data URL\n    before send — check the container log for `failed to inline image` (the\n    inliner deliberately forwards the URL on failure so the backend reports\n    the error). Known cause: CDNs 403 non-descriptive user agents; `images.py`\n    sends a descriptive UA. Also verify the URL itself is valid — a bad URL\n    400s for everyone.\n  - *Base64 `data:` URLs* need no inlining and always work — use one as the\n    regression baseline when debugging the other shapes.\n- **Image generation.** `\u002Fv1\u002Fimages\u002Fgenerations` is served by\n  `provider.py:aimage_generation` via the hosted `image_generation` tool on\n  `\u002Fresponses`. `size`\u002F`quality`\u002F`output_format`\u002F`background` forward onto the\n  tool spec (`auto` is dropped; DALL-E-style `standard`\u002F`hd` map to\n  `medium`\u002F`high`); `n` fans out into concurrent runs (capped at 4);\n  results are always `b64_json`. If it returns \"no image data\", dump the raw\n  SSE events — the b64 lives in `response.output_item.done` items of type\n  `image_generation_call`, field `result`.\n  - *\"no healthy deployments\" for image model names*: clients ask for standard\n    names (`gpt-image-*`, `dall-e-3`); genLitellmConfig aliases those to the\n    top-priority codex slug with `model_info.mode: image_generation`.\n  - *`n`\u002F`quality`\u002F`size` ignored*: they must survive litellm's\n    `get_optional_params_image_gen`, which drops them for custom providers —\n    that's what the shim patch (see Architecture map) fixes. Verify by calling\n    the provider directly in the container with `optional_params={\"n\": 2}`.\n\n## Step 3: Reproduce and fix in the fork\n\n1. Read the current pin from `pkgs\u002Flitellm-codex-oauth-provider.nix` (`rev`).\n2. Clone and check out that exact commit:\n   ```bash\n   d=$(mktemp -d); git clone git@github.com:mkg20001\u002Flitellm-codex-oauth-provider \"$d\u002Ff\"\n   cd \"$d\u002Ff\" && git checkout \u003Cpinned-rev>\n   ```\n3. Locate the failing translation in the layer from Step 2. Prefer OpenAI's\n   typed models (`openai.types.responses`) as the source of truth for the wire\n   shape.\n4. Make the **minimal** fix. Keep pure functions pure (the adapter is\n   deliberately side-effect free). Add a focused helper rather than inlining.\n5. Sanity-check the logic in plain Python before building (fast feedback).\n\nTo probe what the backend itself supports (new hosted tools, content shapes),\nPOST to it directly — copy `\u002Froot\u002F.codex\u002Fauth.json` and\n`\u002Fvar\u002Flib\u002Flitellm-codex\u002Fclient-version.env` off the host, then send a raw\n`\u002Fresponses` payload with headers `Authorization: Bearer \u003Caccess_token>`,\n`chatgpt-account-id`, `OpenAI-Beta: responses=experimental`,\n`originator: codex_cli_rs`, and `version: \u003Cclient_version>` (without `version`\nnewer models 404). This is how image-generation support was discovered: a\npayload with `\"tools\": [{\"type\": \"image_generation\"}]` streamed\n`image_generation_call` items. The container's python also works for quick\nruntime checks: `podman exec litellm sh -c 'cd \u002Fapp && .venv\u002Fbin\u002Fpython -c ...'`\n(the host itself has no python3).\n\n## Step 4: Push to the fork\n\n`origin\u002Fmain` should equal the pinned rev (this repo tracks the fork tip). Commit\nonto `main` and push over SSH (gh here uses the SSH protocol):\n\n```bash\ncd \"$d\u002Ff\"\ngit checkout -B main origin\u002Fmain   # after: git fetch origin main\ngit add -A && git commit -F - \u003C\u003C'EOF'\nfix(\u003Carea>): \u003Cwhat>\n\n\u003Cwhy — include the exact backend error text>\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003Cnoreply@anthropic.com>\nEOF\ngit remote set-url origin git@github.com:mkg20001\u002Flitellm-codex-oauth-provider.git\ngit push origin main\n```\n\nIf `origin\u002Fmain` has advanced past the pinned rev, rebase the fix onto `main`\nfirst so you don't regress newer commits.\n\n## Step 5: Repin the flake\n\nUpdate `pkgs\u002Flitellm-codex-oauth-provider.nix`: bump `rev` to the new commit,\n`version` to `0.3.1-unstable-\u003Ctoday>`, and refresh `hash`:\n\n```bash\nnix-prefetch-url --unpack --type sha256 \\\n  \"https:\u002F\u002Fgithub.com\u002Fmkg20001\u002Flitellm-codex-oauth-provider\u002Farchive\u002F\u003Cnew-rev>.tar.gz\" \\\n  | tail -1 | xargs nix hash to-sri --type sha256\n```\n\nVerify the repinned package builds and contains the fix (the overlay package is\nnot a flake output; build it via the overlay):\n\n```bash\nnix build --impure --no-link --print-out-paths --expr \\\n  'let f = builtins.getFlake (toString .\u002F.); pkgs = import f.inputs.nixpkgs {\n     system = \"x86_64-linux\"; overlays = [ (import .\u002Fpkgs\u002Foverlay.nix) ]; };\n   in pkgs.litellm-codex-oauth-provider'\n```\n\nGrep the resulting store path's\n`lib\u002Fpython*\u002Fsite-packages\u002Flitellm_codex_oauth_provider\u002F*.py` to confirm your\nchange is present.\n\n> If you can't push to the fork (no access), fall back to a local Nix patch:\n> add `patches = [ .\u002Flitellm-codex-oauth-provider-\u003Cslug>.patch ]` to the\n> derivation and stage the patch file (flakes only see git-tracked files).\n> Prefer the fork push; drop the patch once the fork carries the fix.\n\n## Step 6: Deploy\n\n```bash\ngit add pkgs\u002Flitellm-codex-oauth-provider.nix   # flakes only see tracked files\nsh codex.sh   # nixos-rebuild switch --target-host root@codex.plan.ai (impure)\n```\n\nThe build reads `private\u002F*.nix` **through git** (via `private.nix`,\n`\"${builtins.getEnv \"PWD\"}\u002Fprivate\"`), so it needs `git submodule update --init`\nfirst — without it evaluation fails at the host's `imports` with\n`path '\u002Fnix\u002Fstore\u002F…-source\u002Fcodex' does not exist`.\n\nFrom **reagent**, launch the deploy detached instead of waiting on it in the\nforeground; a plain `&` from an interactive terminal gets stopped by SIGTTOU\n(`batch`) and hangs mid-copy. It takes a few minutes — poll the log:\n\n```bash\nsetsid sh -c 'sh codex.sh > \u002Ftmp\u002Fdeploy-codex.log 2>&1; echo EXIT=$? >> \u002Ftmp\u002Fdeploy-codex.log' \\\n  \u003C \u002Fdev\u002Fnull &                       # from a devshell: false exec\ntail -n 3 \u002Ftmp\u002Fdeploy-codex.log       # plain `tail -3` is refused here\n```\n\n`codex.sh` regenerates the config dir (`litellm-codex-config`) and restarts the\ncontainer on switch — but only if the container\u002Fconfig **unit text** changed.\nA provider-source-only bump changes the static dir the units reference, yet\nthe switch output may not list `podman-litellm` under \"restarting\". If it\ndidn't restart, do it explicitly (this also re-runs the config oneshot):\n\n```bash\nssh root@codex.plan.ai 'systemctl restart podman-litellm.service'\n```\n\n## Step 7: Verify\n\n```bash\nssh root@codex.plan.ai 'journalctl -u podman-litellm --no-pager -n 80'\n```\n\nConfirm the container came up healthy and the original error is gone. Ideally\nre-trigger the failing request (or ask the user to) and watch the log clear.\n\nTest end-to-end through the proxy with the master key (in `\u002Fetc\u002Flitellm.env`\non the host — `ssh root@codex.plan.ai 'grep LITELLM_MASTER_KEY \u002Fetc\u002Flitellm.env'`).\nThe proxy takes ~30s to come up after a restart; poll `\u002Fv1\u002Fmodels` until 200.\n\n```bash\nKEY=sk-...; BASE=https:\u002F\u002Fcodex.plan.ai\n# capabilities litellm advertises (drives client behavior)\ncurl -s $BASE\u002Fmodel\u002Finfo -H \"Authorization: Bearer $KEY\"\n# chat (add \"stream\": true to cover the astreaming path too)\ncurl -s $BASE\u002Fv1\u002Fchat\u002Fcompletions -H \"Authorization: Bearer $KEY\" \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\"model\":\"\u003Cslug>\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'\n# vision: content part {\"type\":\"image_url\",\"image_url\":{\"url\":\"\u003Cdata-or-https-url>\"}}\n# image generation:\ncurl -s $BASE\u002Fv1\u002Fimages\u002Fgenerations -H \"Authorization: Bearer $KEY\" \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\"model\":\"\u003Cslug>\",\"prompt\":\"...\",\"size\":\"1024x1024\",\"quality\":\"low\"}'\n```\n\nFor vision, test both a base64 `data:` URL (pure translation path) and a\nremote `https:` URL (exercises the `images.py` inliner).\n\n## Step 8: Commit\n\nCommit the repin with a conventional message; push only if the user asks.\n\n```bash\ngit commit -m 'fix(codex): bump litellm-codex-oauth-provider — \u003Cwhat>'\n```\n\nDo not touch the unrelated pre-existing `flake.lock` modification unless asked.\n\n## Common pitfalls\n\n1. Flakes only include **git-tracked** files — `git add` any new patch file or\n   the deploy build won't see it.\n2. The overlay package isn't a flake output; build it via the overlay expression\n   above, not `nix build .#litellm-codex-oauth-provider`.\n3. Codex model slugs are discovered live and change over time — never hardcode\n   them; regenerate the config instead.\n4. Keep `adapter.py` helpers side-effect free (sync + async paths share them).\n5. Request bugs → `prompts.py`; response bugs → `adapter.py`. The 400's `param`\n   (`input[...]` = request) tells you which.\n6. gh here pushes over SSH — if an https remote fails with \"could not read\n   Username\", `git remote set-url origin git@github.com:...`.\n7. After `sh codex.sh`, check the switch output actually restarted\n   `podman-litellm` — a provider-only bump may not, leaving the old code\n   running (see Step 6).\n8. The host has no `python3` on PATH — run helper scripts locally or inside\n   the container (`podman exec litellm ... .venv\u002Fbin\u002Fpython`).\n9. When fetching third-party URLs from the provider (e.g. image inlining),\n   send a descriptive `User-Agent` — CDNs like Wikimedia 403 library-default\n   UAs.\n10. When driving this from reagent, run shell commands with `devshell: false`\n   (no default devshell in this flake) and `ssh-keyscan` the host into\n   `~\u002F.ssh\u002Fknown_hosts` before the first ssh — see Step 1.\n11. Deploy from reagent with the detached `setsid sh -c 'sh codex.sh > … &'`\n   form, not a foreground `sh codex.sh` and not a plain `&` (which SIGTTOUs and\n   hangs mid-copy) — see Step 6. `private\u002F` must be initialised\n   (`git submodule update --init`) or evaluating\u002Fbuilding the host can't read\n   `private\u002F*.nix`.\n\n## Verification checklist\n\n- [ ] captured the exact error and identified the layer (request\u002Fresponse\u002Fmodel\u002Fauth)\n- [ ] fix committed and pushed to the fork's `main`\n- [ ] `pkgs\u002Flitellm-codex-oauth-provider.nix` repinned (rev + hash + version)\n- [ ] repinned package builds and contains the fix\n- [ ] `sh codex.sh` deployed successfully\n- [ ] container healthy and original error gone in the logs\n- [ ] repin committed with a conventional message\n","# Fix codex.plan.ai litellm plugin\n\n## Overview\n\n`codex.plan.ai` runs a litellm proxy (official `ghcr.io\u002Fberriai\u002Flitellm-database`\nimage, DB mode) fronting ChatGPT's Codex backend through a **custom OAuth\nprovider**. The provider is a pinned fork —\n`mkg20001\u002Flitellm-codex-oauth-provider` — that bridges chat-completions calls to\nthe Codex `\u002Fresponses` API (`https:\u002F\u002Fchatgpt.com\u002Fbackend-api\u002Fcodex\u002Fresponses`).\n\nErrors surface in the calling app's logs (e.g. `openclaw` on other hosts) and in\nthe litellm container logs on `codex.plan.ai`. Most bugs live in the **fork's\ntranslation code**, not in this repo's Nix. This skill diagnoses the failure,\nfixes it at the right layer, and redeploys.\n\nRun all repo commands from the `nixos-infra` root. The git\u002Fgh account here is\n`mkg20001`, which has **ADMIN** on the fork — fixes go directly to the fork's\n`main`.\n\nFor a **check** rather than a repair — is the proxy serving right now, and which\nlayer is down — use the `codex-healthcheck` skill first; it probes\n`\u002Fv1\u002Fmodels` + one tiny chat completion and only sends you here for the deep\nlayer. Come back to this skill for the diagnosis, the fork fix and the redeploy.\n\n## Architecture map\n\n- `codex\u002Fdefault.nix` — the host. Builds the litellm config dir at each start\n  (`litellm-codex-config` oneshot → `genLitellmConfig`), runs the container, and\n  defines the `codex_handler.py` shim. Config dir lives at `\u002Fvar\u002Flib\u002Flitellm-codex`,\n  bind-mounted read-only into `\u002Fconfig`.\n- `pkgs\u002Flitellm-codex-oauth-provider.nix` — packages the fork (pinned `rev` +\n  `hash`). Its source is copied into the config dir (importable via\n  `PYTHONPATH=\u002Fconfig`).\n- Fork internals (clone to inspect — see Step 3):\n  - `prompts.py` — **request** side: chat-completions messages → Codex `input`.\n    `derive_instructions`, `_to_codex_input_items`, `_to_responses_content`.\n  - `adapter.py` — **response** side: Codex `\u002Fresponses` payload (JSON or SSE) →\n    litellm `ModelResponse`. `transform_response`, `convert_sse_to_json`.\n  - `models.py` \u002F `model_map.py` — live model discovery + slug mapping.\n  - `auth.py` — OAuth access-token refresh (rewrites `\u002Froot\u002F.codex\u002Fauth.json`).\n  - `streaming_utils.py`, `sse_utils.py` — SSE handling.\n  - `images.py` — vision: downloads remote `http(s)` image URLs and inlines\n    them as base64 `data:` URLs (the backend can't reliably fetch arbitrary\n    public URLs — the Codex CLI only ever sends data URLs).\n  - `provider.py` also implements `image_generation`\u002F`aimage_generation` on\n    top of the Responses-API hosted `image_generation` tool (there is no\n    dedicated images endpoint; the b64 image arrives in\n    `image_generation_call` output items' `result` field).\n- Handler shim (`codexHandlerPy` in `codex\u002Fdefault.nix`) — re-exports the\n  provider instance AND removes discovered model slugs from\n  `litellm.open_ai_chat_completion_models` so litellm's dispatch doesn't route\n  codex models to its built-in OpenAI handler. Also patches\n  `litellm.images.main.get_optional_params_image_gen`: for custom providers\n  litellm returns `{}` there (its provider-config registry is a hardcoded\n  if\u002Felif over built-ins), silently dropping standard image params\n  (`n`\u002F`quality`\u002F`size`\u002F`style`) before they reach the custom handler.\n\n## Step 1: Capture the error\n\nGet the exact error and its shape. The `param`\u002Ftraceback tells you which layer.\n\nAsk the user for the failing log line if not already provided, and pull the\ncontainer logs from the host:\n\n```bash\nssh root@codex.plan.ai 'journalctl -u podman-litellm --no-pager -n 200'\nssh root@codex.plan.ai 'journalctl -u litellm-codex-config --no-pager -n 80'\n```\n\nWhen you drive this from **reagent** (rather than an interactive terminal) on\nthis project: run shell tasks with `devshell: false` — the flake has no default\ndevshell and reagent's automatic `nix develop` wrapper fails there. Run\n`ssh-keyscan codex.plan.ai >> ~\u002F.ssh\u002Fknown_hosts` once (and for any other host\nyou deploy to) before the first ssh, or the host key prompt eats the command.\n\n`private\u002F` is a git submodule and `private\u002F*.nix` (e.g. `private\u002Fcodex.nix`) is\nread from it — run `git submodule update --init` on a fresh checkout, or the\nbuild fails (see Step 6).\n\n## Step 2: Classify the failure\n\n- **Request-translation (most common).** HTTP 400 from `...\u002Fcodex\u002Fresponses`\n  with `invalid_request_error` naming an `input[...]` param, e.g.\n  `Invalid value: 'text' ... Supported values are: 'input_text', ...` at\n  `input[N].content[0].type`. → fix `prompts.py`. (Known example: list-form\n  message `content` parts forwarded with chat-completions types instead of\n  Responses-API types `input_text`\u002F`input_image`\u002F`output_text`.)\n- **Response-translation.** Errors after a 200 from Codex — missing choices,\n  bad tool_calls, SSE parse failures, wrong usage. → fix `adapter.py`.\n- **Model not supported.** `model is not supported when using Codex with a\n  ChatGPT account`, or a slug that 400s. Codex slugs change over time and are\n  **discovered live** — nothing is hardcoded. Regenerate the config:\n  `ssh root@codex.plan.ai systemctl restart podman-litellm` (re-runs\n  `litellm-codex-config`). Check its journal for the discovered model_list.\n- **Missing\u002Fhidden models, or 404 \"Model not found\" on \u002Fresponses.** The\n  backend version-gates BOTH endpoints: `\u002Fcodex\u002Fmodels` by the `client_version`\n  query param and `\u002Fcodex\u002Fresponses` by the `version` header. genLitellmConfig\n  resolves the latest codex release tag from GitHub at each start and shares it\n  via `\u002Fvar\u002Flib\u002Flitellm-codex\u002Fclient-version.env` (fallback:\n  `pkgs.codex.version`, which lags upstream). If new models are missing, check\n  the journal's `codex client_version:` line first.\n- **Broken\u002Fempty pricing (`priced models:` empty in the config journal).**\n  Prices are scraped from the OpenAI pricing page by regex in\n  `codex\u002Fdefault.nix` (`fetch_pricing`); the page's encoded-row format has\n  changed more than once (column added, `\"-\"` placeholders for missing prices).\n  Fetch the page, inspect the encoded rows around a known slug, adjust the\n  parser, and test it standalone before deploying.\n- **Dispatch \u002F \"api_key must be set\".** Error names the custom provider\n  (`CodexException`) but the traceback is in `llms\u002Fopenai\u002Fopenai.py` — a codex\n  slug leaked into `litellm.open_ai_chat_completion_models`. → the shim in\n  `codex\u002Fdefault.nix` (`codexHandlerPy`) should strip it; verify\n  `available_model_slugs()` covers the slug.\n- **Auth \u002F token.** 401\u002Fexpired. The provider refreshes `\u002Froot\u002F.codex\u002Fauth.json`\n  in place (mounted rw). If refresh is broken, re-run `codex login` on the host.\n  Fixes to refresh logic go in `auth.py`.\n- **Vision \u002F images.** Three distinct shapes:\n  - *Client refuses to send images* (\"model does not support vision\"): litellm\n    only advertises what `model_info` declares. genLitellmConfig derives it per\n    model from the discovery payload (`supports_vision` ← `input_modalities`\n    contains `\"image\"`, `max_input_tokens` ← `context_window`); check\n    `\u002Fmodel\u002Finfo` on the proxy first.\n  - *Remote image URL 400s* with `\"Error while downloading file. Upstream\n    status code: NNN.\"`, param `url`: the backend tried (and failed) to fetch\n    the URL server-side. `images.py` should have inlined it as a data URL\n    before send — check the container log for `failed to inline image` (the\n    inliner deliberately forwards the URL on failure so the backend reports\n    the error). Known cause: CDNs 403 non-descriptive user agents; `images.py`\n    sends a descriptive UA. Also verify the URL itself is valid — a bad URL\n    400s for everyone.\n  - *Base64 `data:` URLs* need no inlining and always work — use one as the\n    regression baseline when debugging the other shapes.\n- **Image generation.** `\u002Fv1\u002Fimages\u002Fgenerations` is served by\n  `provider.py:aimage_generation` via the hosted `image_generation` tool on\n  `\u002Fresponses`. `size`\u002F`quality`\u002F`output_format`\u002F`background` forward onto the\n  tool spec (`auto` is dropped; DALL-E-style `standard`\u002F`hd` map to\n  `medium`\u002F`high`); `n` fans out into concurrent runs (capped at 4);\n  results are always `b64_json`. If it returns \"no image data\", dump the raw\n  SSE events — the b64 lives in `response.output_item.done` items of type\n  `image_generation_call`, field `result`.\n  - *\"no healthy deployments\" for image model names*: clients ask for standard\n    names (`gpt-image-*`, `dall-e-3`); genLitellmConfig aliases those to the\n    top-priority codex slug with `model_info.mode: image_generation`.\n  - *`n`\u002F`quality`\u002F`size` ignored*: they must survive litellm's\n    `get_optional_params_image_gen`, which drops them for custom providers —\n    that's what the shim patch (see Architecture map) fixes. Verify by calling\n    the provider directly in the container with `optional_params={\"n\": 2}`.\n\n## Step 3: Reproduce and fix in the fork\n\n1. Read the current pin from `pkgs\u002Flitellm-codex-oauth-provider.nix` (`rev`).\n2. Clone and check out that exact commit:\n   ```bash\n   d=$(mktemp -d); git clone git@github.com:mkg20001\u002Flitellm-codex-oauth-provider \"$d\u002Ff\"\n   cd \"$d\u002Ff\" && git checkout \u003Cpinned-rev>\n   ```\n3. Locate the failing translation in the layer from Step 2. Prefer OpenAI's\n   typed models (`openai.types.responses`) as the source of truth for the wire\n   shape.\n4. Make the **minimal** fix. Keep pure functions pure (the adapter is\n   deliberately side-effect free). Add a focused helper rather than inlining.\n5. Sanity-check the logic in plain Python before building (fast feedback).\n\nTo probe what the backend itself supports (new hosted tools, content shapes),\nPOST to it directly — copy `\u002Froot\u002F.codex\u002Fauth.json` and\n`\u002Fvar\u002Flib\u002Flitellm-codex\u002Fclient-version.env` off the host, then send a raw\n`\u002Fresponses` payload with headers `Authorization: Bearer \u003Caccess_token>`,\n`chatgpt-account-id`, `OpenAI-Beta: responses=experimental`,\n`originator: codex_cli_rs`, and `version: \u003Cclient_version>` (without `version`\nnewer models 404). This is how image-generation support was discovered: a\npayload with `\"tools\": [{\"type\": \"image_generation\"}]` streamed\n`image_generation_call` items. The container's python also works for quick\nruntime checks: `podman exec litellm sh -c 'cd \u002Fapp && .venv\u002Fbin\u002Fpython -c ...'`\n(the host itself has no python3).\n\n## Step 4: Push to the fork\n\n`origin\u002Fmain` should equal the pinned rev (this repo tracks the fork tip). Commit\nonto `main` and push over SSH (gh here uses the SSH protocol):\n\n```bash\ncd \"$d\u002Ff\"\ngit checkout -B main origin\u002Fmain   # after: git fetch origin main\ngit add -A && git commit -F - \u003C\u003C'EOF'\nfix(\u003Carea>): \u003Cwhat>\n\n\u003Cwhy — include the exact backend error text>\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003Cnoreply@anthropic.com>\nEOF\ngit remote set-url origin git@github.com:mkg20001\u002Flitellm-codex-oauth-provider.git\ngit push origin main\n```\n\nIf `origin\u002Fmain` has advanced past the pinned rev, rebase the fix onto `main`\nfirst so you don't regress newer commits.\n\n## Step 5: Repin the flake\n\nUpdate `pkgs\u002Flitellm-codex-oauth-provider.nix`: bump `rev` to the new commit,\n`version` to `0.3.1-unstable-\u003Ctoday>`, and refresh `hash`:\n\n```bash\nnix-prefetch-url --unpack --type sha256 \\\n  \"https:\u002F\u002Fgithub.com\u002Fmkg20001\u002Flitellm-codex-oauth-provider\u002Farchive\u002F\u003Cnew-rev>.tar.gz\" \\\n  | tail -1 | xargs nix hash to-sri --type sha256\n```\n\nVerify the repinned package builds and contains the fix (the overlay package is\nnot a flake output; build it via the overlay):\n\n```bash\nnix build --impure --no-link --print-out-paths --expr \\\n  'let f = builtins.getFlake (toString .\u002F.); pkgs = import f.inputs.nixpkgs {\n     system = \"x86_64-linux\"; overlays = [ (import .\u002Fpkgs\u002Foverlay.nix) ]; };\n   in pkgs.litellm-codex-oauth-provider'\n```\n\nGrep the resulting store path's\n`lib\u002Fpython*\u002Fsite-packages\u002Flitellm_codex_oauth_provider\u002F*.py` to confirm your\nchange is present.\n\n> If you can't push to the fork (no access), fall back to a local Nix patch:\n> add `patches = [ .\u002Flitellm-codex-oauth-provider-\u003Cslug>.patch ]` to the\n> derivation and stage the patch file (flakes only see git-tracked files).\n> Prefer the fork push; drop the patch once the fork carries the fix.\n\n## Step 6: Deploy\n\n```bash\ngit add pkgs\u002Flitellm-codex-oauth-provider.nix   # flakes only see tracked files\nsh codex.sh   # nixos-rebuild switch --target-host root@codex.plan.ai (impure)\n```\n\nThe build reads `private\u002F*.nix` **through git** (via `private.nix`,\n`\"${builtins.getEnv \"PWD\"}\u002Fprivate\"`), so it needs `git submodule update --init`\nfirst — without it evaluation fails at the host's `imports` with\n`path '\u002Fnix\u002Fstore\u002F…-source\u002Fcodex' does not exist`.\n\nFrom **reagent**, launch the deploy detached instead of waiting on it in the\nforeground; a plain `&` from an interactive terminal gets stopped by SIGTTOU\n(`batch`) and hangs mid-copy. It takes a few minutes — poll the log:\n\n```bash\nsetsid sh -c 'sh codex.sh > \u002Ftmp\u002Fdeploy-codex.log 2>&1; echo EXIT=$? >> \u002Ftmp\u002Fdeploy-codex.log' \\\n  \u003C \u002Fdev\u002Fnull &                       # from a devshell: false exec\ntail -n 3 \u002Ftmp\u002Fdeploy-codex.log       # plain `tail -3` is refused here\n```\n\n`codex.sh` regenerates the config dir (`litellm-codex-config`) and restarts the\ncontainer on switch — but only if the container\u002Fconfig **unit text** changed.\nA provider-source-only bump changes the static dir the units reference, yet\nthe switch output may not list `podman-litellm` under \"restarting\". If it\ndidn't restart, do it explicitly (this also re-runs the config oneshot):\n\n```bash\nssh root@codex.plan.ai 'systemctl restart podman-litellm.service'\n```\n\n## Step 7: Verify\n\n```bash\nssh root@codex.plan.ai 'journalctl -u podman-litellm --no-pager -n 80'\n```\n\nConfirm the container came up healthy and the original error is gone. Ideally\nre-trigger the failing request (or ask the user to) and watch the log clear.\n\nTest end-to-end through the proxy with the master key (in `\u002Fetc\u002Flitellm.env`\non the host — `ssh root@codex.plan.ai 'grep LITELLM_MASTER_KEY \u002Fetc\u002Flitellm.env'`).\nThe proxy takes ~30s to come up after a restart; poll `\u002Fv1\u002Fmodels` until 200.\n\n```bash\nKEY=sk-...; BASE=https:\u002F\u002Fcodex.plan.ai\n# capabilities litellm advertises (drives client behavior)\ncurl -s $BASE\u002Fmodel\u002Finfo -H \"Authorization: Bearer $KEY\"\n# chat (add \"stream\": true to cover the astreaming path too)\ncurl -s $BASE\u002Fv1\u002Fchat\u002Fcompletions -H \"Authorization: Bearer $KEY\" \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\"model\":\"\u003Cslug>\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'\n# vision: content part {\"type\":\"image_url\",\"image_url\":{\"url\":\"\u003Cdata-or-https-url>\"}}\n# image generation:\ncurl -s $BASE\u002Fv1\u002Fimages\u002Fgenerations -H \"Authorization: Bearer $KEY\" \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\"model\":\"\u003Cslug>\",\"prompt\":\"...\",\"size\":\"1024x1024\",\"quality\":\"low\"}'\n```\n\nFor vision, test both a base64 `data:` URL (pure translation path) and a\nremote `https:` URL (exercises the `images.py` inliner).\n\n## Step 8: Commit\n\nCommit the repin with a conventional message; push only if the user asks.\n\n```bash\ngit commit -m 'fix(codex): bump litellm-codex-oauth-provider — \u003Cwhat>'\n```\n\nDo not touch the unrelated pre-existing `flake.lock` modification unless asked.\n\n## Common pitfalls\n\n1. Flakes only include **git-tracked** files — `git add` any new patch file or\n   the deploy build won't see it.\n2. The overlay package isn't a flake output; build it via the overlay expression\n   above, not `nix build .#litellm-codex-oauth-provider`.\n3. Codex model slugs are discovered live and change over time — never hardcode\n   them; regenerate the config instead.\n4. Keep `adapter.py` helpers side-effect free (sync + async paths share them).\n5. Request bugs → `prompts.py`; response bugs → `adapter.py`. The 400's `param`\n   (`input[...]` = request) tells you which.\n6. gh here pushes over SSH — if an https remote fails with \"could not read\n   Username\", `git remote set-url origin git@github.com:...`.\n7. After `sh codex.sh`, check the switch output actually restarted\n   `podman-litellm` — a provider-only bump may not, leaving the old code\n   running (see Step 6).\n8. The host has no `python3` on PATH — run helper scripts locally or inside\n   the container (`podman exec litellm ... .venv\u002Fbin\u002Fpython`).\n9. When fetching third-party URLs from the provider (e.g. image inlining),\n   send a descriptive `User-Agent` — CDNs like Wikimedia 403 library-default\n   UAs.\n10. When driving this from reagent, run shell commands with `devshell: false`\n   (no default devshell in this flake) and `ssh-keyscan` the host into\n   `~\u002F.ssh\u002Fknown_hosts` before the first ssh — see Step 1.\n11. Deploy from reagent with the detached `setsid sh -c 'sh codex.sh > … &'`\n   form, not a foreground `sh codex.sh` and not a plain `&` (which SIGTTOUs and\n   hangs mid-copy) — see Step 6. `private\u002F` must be initialised\n   (`git submodule update --init`) or evaluating\u002Fbuilding the host can't read\n   `private\u002F*.nix`.\n\n## Verification checklist\n\n- [ ] captured the exact error and identified the layer (request\u002Fresponse\u002Fmodel\u002Fauth)\n- [ ] fix committed and pushed to the fork's `main`\n- [ ] `pkgs\u002Flitellm-codex-oauth-provider.nix` repinned (rev + hash + version)\n- [ ] repinned package builds and contains the fix\n- [ ] `sh codex.sh` deployed successfully\n- [ ] container healthy and original error gone in the logs\n- [ ] repin committed with a conventional message\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"description":7,"user_invocable":36},"plan.ai","1.0.0",{"hermes":27},{"tags":28,"related_skills":34},[29,30,31,32,33],"nixos","infrastructure","litellm","codex","deploy",[35],"update-nixos",true,"trunk","ae6b6d8ae743bf5a1af509d00f07168869ab26b3",".claude\u002Fskills\u002Ffix-codex-litellm",0,"https:\u002F\u002Fgithub.com\u002Fplanailabs\u002Fnixos-infra","https:\u002F\u002Fgithub.com\u002Fplanailabs","https:\u002F\u002Fplan.ai","2026-11-13 18:32:11",{"source":8,"skillName":5,"commands":46,"prompt":56,"zipAvailable":36},[47,50,53],{"tool":48,"command":49},"npx","npx skills add planailabs\u002Fnixos-infra --skill fix-codex-litellm",{"tool":51,"command":52},"bunx","bunx skills add planailabs\u002Fnixos-infra --skill fix-codex-litellm",{"tool":54,"command":55},"pnpm","pnpm dlx skills add planailabs\u002Fnixos-infra --skill fix-codex-litellm","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fplanailabs\u002Fnixos-infra\u002Fblob\u002Ftrunk\u002F.claude\u002Fskills\u002Ffix-codex-litellm\u002FSKILL.md",[],[59,64,70,77],{"skillNo":60,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":61,"categories":62,"keywords":63,"publishedAt":19},"01M784SR5XNTH2VERG3XDJENK2","https:\u002F\u002Fgithub.com\u002Fplanailabs\u002Fnixos-infra\u002Fblob\u002Ftrunk\u002F.hermes\u002Fskills\u002Ffix-codex-litellm\u002FSKILL.md",[],[],{"skillNo":65,"name":35,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M784SJ9C733C0G2GFXR5EJ5C","Use when updating NixOS flake inputs and deploying nixos-infra to all plan.ai servers, including fixing build errors and pushing resulting commits.","https:\u002F\u002Fgithub.com\u002Fplanailabs\u002Fnixos-infra\u002Fblob\u002Ftrunk\u002F.claude\u002Fskills\u002Fupdate-nixos\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M784S8HE4XHFBK0MFZ4VTFX6","zitadel-bootstrap","Use when setting up Zitadel for a new plan.ai-style deployment (a cloud console, a chat\u002Fagent and optionally a Harbor on their own domains, like webapp.agency) or re-running it — asks for the domains, org and login names, social-login credentials and shared keys, writes ansible\u002Fzitadel\u002Fdeployments\u002F\u003Cname>.yml, runs setup.yml (Zitadel orgs, login policy, IdPs, projects, clients, service users) and secrets.yml (the rest of the secrets, private\u002F\u003Cname>-{cloud,chat,harbor}.nix).","https:\u002F\u002Fgithub.com\u002Fplanailabs\u002Fnixos-infra\u002Fblob\u002Ftrunk\u002F.agents\u002Fskills\u002Fzitadel-bootstrap\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M784S448ZG86SRRW2RQJH65N","codex-healthcheck","Use when checking whether the codex.plan.ai litellm proxy works (scheduled health check) or when it is reported broken — probe \u002Fv1\u002Fmodels + a tiny chat completion, classify the failure against the known layers, then fix it (container\u002Fconfig, fork+repin+deploy, auth) and verify.","https:\u002F\u002Fgithub.com\u002Fplanailabs\u002Fnixos-infra\u002Fblob\u002Ftrunk\u002F.agents\u002Fskills\u002Fcodex-healthcheck\u002FSKILL.md",[],[]]