liter-llm

<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:3e0dcb446dcc37cd4709da048b254902f8d7f825b553308c5d25074419895562 Source-Hash: blake3:725e6663621cbc5ffbaf9dbb2141fe5f9b95aee234fe9f7ece2d2b6e63dc5a66 Schema-Version: v1 -->

Liter-LLM Universal LLM Client

Liter-LLM is a universal LLM API client with a Rust core and native bindings for Python, TypeScript/Node.js, Go, Java, C#, Ruby, PHP, Elixir, WebAssembly, and C (FFI). One unified interface reaches 174 providers (OpenAI, Anthropic, Google Gemini, Groq, Mistral, Cohere, AWS Bedrock, Azure, and many more).

Capability map

• Modalities — chat completions, streaming, tool/function calling, structured outputs, embeddings, image generation, audio (speech, transcription), moderation, web search, OCR, reranking, batch, and file operations.
• Provider routing — provider/model prefix selects the backend (openai/gpt-4o, anthropic/claude-sonnet-4-20250514, gemini/gemini-2.0-flash). Set a full provider/model as model_hint to choose the client provider.
• Middleware — response caching, rate limiting (RPM/TPM), cost tracking, budget enforcement (hard / soft), fallback chains, circuit-breaker cooldown, and background health checks.
• Search providers (12) and OCR providers (4) — first-class search and ocr methods routed by the same prefix convention.
• Proxy server — liter-llm api exposes an OpenAI-compatible gateway with 22 REST endpoints, virtual API keys, budgets, and SSE streaming.
• MCP server — liter-llm mcp exposes 22 tools mirroring the proxy endpoints, for MCP-compatible clients (Claude Code, Claude Desktop).
• 14 language bindings — same surface, language-native naming (snake_case for Python/Rust/Ruby/Go/Elixir/PHP, camelCase for TS/Node/WASM/C#/Java).

When to use the MCP server vs the SDK vs the proxy

Use the …	When you want to …	Entry point
MCP server	Let the agent call LLM APIs directly as tools, with no glue code	liter-llm mcp --transport stdio (this plugin auto-registers it)
SDK / binding	Write application code that calls LLMs in a specific language	pip install liter-llm, cargo add liter-llm, etc.
Proxy	Give many apps/teams a shared OpenAI-compatible endpoint with keys, budgets, and rate limits	liter-llm api --config liter-llm-proxy.toml

Rule of thumb: reach for the MCP server inside an agent session; the SDK when building software; the proxy when centralizing access for multiple consumers.

Installation

CLI (proxy + MCP server)

# Homebrew (macOS / Linux)
brew install xberg-io/tap/liter-llm

# or run it without a persistent install (the CLI proxy package self-installs the binary)
pnpm dlx @xberg-io/liter-llm-cli@2.2.3 --help

# or download a prebuilt binary from the latest GitHub release:
#   https://github.com/xberg-io/liter-llm/releases/latest

# or build from source
cargo install liter-llm-cli

# or Docker (35MB image)
docker pull ghcr.io/xberg-io/liter-llm

Language bindings

Language	Install
Python	pip install liter-llm
Node.js	pnpm add @xberg-io/liter-llm
Rust	cargo add liter-llm
Go	go get github.com/xberg-io/liter-llm/packages/go/v2
Ruby	gem install liter_llm
PHP	composer require xberg-io/liter-llm
C#	dotnet add package XbergIo.LiterLlm
WASM	pnpm add @xberg-io/liter-llm-wasm

Quick start (Python, async)

import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = ChatCompletionRequest.from_json(
        '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'
    )
    response = await client.chat(request)
    print(response.choices[0].message.content)

asyncio.run(main())

Provider routing

The prefix before / in the request's model selects the provider:

ChatCompletionRequest.from_json('{"model":"openai/gpt-4o","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')

Client construction resolves credentials for its selected provider from environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY, and AWS credentials for Bedrock. Create a separate client for each provider with matching model_hint and credentials. Request prefixes change routing but reuse the client API key; they do not select a new environment key per request.

Configuration

Pass basic options to create_client(...) (api_key, base_url, timeout_secs, connect_timeout_secs, max_retries, model_hint), or set the full surface in a liter-llm.toml (SDK) / liter-llm-proxy.toml (proxy) — both auto-discover from the cwd upward.

# liter-llm.toml
api_key = "${OPENAI_API_KEY}"
model_hint = "openai/gpt-4o"
timeout_secs = 120
max_retries = 5

[cache]
max_entries = 512
ttl_seconds = 600

[budget]
global_limit = 50.0
enforcement = "hard"

[rate_limit]
rpm = 60
tpm = 100000

Option	Default	Description
api_key	required	Provider key, wrapped in SecretString (never logged).
base_url	from registry	Override the provider base URL.
model_hint	none	Full provider/model used to select the client provider and credentials.
timeout_secs	60	Request timeout in seconds.
connect_timeout_secs	10	Connect timeout in seconds. A connection that does not open in time is retried up to max_retries times. Zero is an error on every target. wasm32 ignores any other value.
max_retries	3	Retries on 429/5xx with exponential backoff.
cache	none	max_entries, ttl_seconds.
budget	none	global_limit, model_limits, enforcement.
rate_limit	none	rpm, tpm.
cost_tracking	false	Per-request cost tracking.
tracing	false	OpenTelemetry spans.

Proxy server

liter-llm api --config liter-llm-proxy.toml

22 OpenAI-compatible endpoints, model routing by prefix, virtual API keys, per-key RPM/TPM limits, cost tracking, budget enforcement, response caching, SSE streaming, and an OpenAPI 3.1 spec at /openapi.json.

MCP server

liter-llm mcp --transport stdio   # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001

This plugin auto-registers the stdio server via scripts/mcp-launch.sh. The 22 tools mirror the proxy endpoints: chat, embed, generate_image, speech, transcribe, moderate, rerank, search, ocr, list_models; file ops (create_file, list_files, retrieve_file, delete_file, file_content); batch ops (create_batch, list_batches, retrieve_batch, cancel_batch); and Responses API (create_response, retrieve_response, cancel_response).

Common pitfalls

1. Provider routing — use "provider/model" to select a request backend. Bare model names fall back to the configured client provider (OpenAI by default).
2. API keys are SecretString — never logged or serialized. Read from env vars; never hardcode.
3. Python methods are async — await inside an async context.
4. Naming conventions differ — camelCase for TS/Node/WASM/C#/Java, snake_case elsewhere.
5. Streaming chunks may have null content — null-check chunk.choices[0].delta.content before use.
6. Budget modes — hard rejects over-budget requests; soft logs and allows.

Additional resources

• Upstream docs: https://docs.liter-llm.xberg.io
• GitHub: https://github.com/xberg-io/liter-llm