promptfoo-provider-setup
Configure promptfoo providers or redteam targets for hosted models, live HTTP APIs, Python/JavaScript local scripts, agent SDKs, or multi-input systems. Use when connecting promptfoo to the system under test, mapping vars, auth env vars, request bodies, response transforms, or static-code-derived provider wrappers. Do not use for choosing eval assertions or red team plugins unless a smoke test is needed to verify the connection.
- License 许可
- Apache-2.0
- 所属仓库
- nuculo/strazh
- 最近更新
- 2026-09-28
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- promptfoo-provider-setup
- description
- Configure promptfoo providers or redteam targets for hosted models, live HTTP APIs, Python/JavaScript local scripts, agent SDKs, or multi-input systems. Use when connecting promptfoo to the system under test, mapping vars, auth env vars, request bodies, response transforms, or static-code-derived provider wrappers. Do not use for choosing eval assertions or red team plugins unless a smoke test is needed to verify the connection.
02 资源文件 3
- 参考文档agents/openai.yaml303 B
- 参考文档references/provider-patterns.md8.5 KB
- 脚本scripts/openapi-operation-to-config.mjs30.8 KB
03 SKILL.md 正文
Promptfoo Provider Setup
Connect Promptfoo to the system under test with the smallest reliable provider or target configuration. Prefer a working smoke test over a clever abstraction.
Read references/provider-patterns.md when you need concrete YAML or provider
wrapper examples.
For OpenAPI specs, you can run the bundled
scripts/openapi-operation-to-config.mjs to draft a one-operation HTTP smoke
config, then inspect and edit the result before probing. The script ships in this
skill's scripts/ directory; when the skill is installed as a plugin it lives in
the plugin cache, not your project, so run it by its absolute path (or copy it in)
rather than a bare scripts/... path. With --token-env, it
infers Bearer/OAuth2/OpenID and header/query/cookie API-key auth; use
--auth-header/--auth-prefix to override.
Inputs
Infer from the repo or user prompt when possible:
- Target surface: hosted model, live HTTP endpoint, local function/script, agent harness, MCP/tool agent, or redteam target.
- Invocation shape: method, URL/path, headers, request body, input vars, auth, streaming/statefulness, and expected response field.
- Safety boundary: whether it is okay to call the live endpoint and which sample payload is safe.
- Output goal: eval provider block, redteam
targetsblock, local provider wrapper, or a minimal smoke-test suite.
If the contract is unclear, create a conservative TODO-marked starter and state exactly what must be verified before using it against production.
Workflow
1. Pick discovery mode
Use one of these modes, or combine them:
- Live HTTP endpoint: probe an already-running endpoint with safe requests.
- Static code discovery: inspect route handlers, OpenAPI specs, tests, SDK clients, or existing fetch/axios calls.
- Hybrid: compare static contract assumptions with a live probe.
- Wrapper mode: write
provider.jsorprovider.pywhen built-in providers cannot express auth, signing, streaming, multi-step calls, or custom parsing.
Do not send secrets to unknown endpoints. Use {{env.VAR}} placeholders in
configs and local environment variables only in shell commands.
2. Discover the contract
For live HTTP endpoints:
- Start with non-mutating checks: docs URL, OpenAPI URL, health endpoint,
OPTIONS, or a safeGET. - Make at most one safe representative call before writing config.
- Capture the response shape and status/error behavior.
- Prefer explicit JSON paths in
transformResponse, such asjson.output. - Use
queryParamsfor query-string fields on any HTTP method, and use thetextvariable intransformResponsefor plain-text responses. - Set
stateful: falsefor stateless endpoints; otherwisevalidate targetwill run a session-memory check. For stateful apps, include{{sessionId}}in the request or configure server-side session parsing.
For static code discovery:
- Search for route definitions, tests, and clients with
rg. - Identify method, path, required headers, request schema, response schema, and authentication source.
- If the app constructs prompts dynamically, wrap the real code instead of duplicating business logic in YAML.
- For agents/tools, identify whether Promptfoo should send one string input or a structured object with named fields.
3. Choose the provider pattern
- Use
id: httpsfor straightforward JSON HTTP APIs. - Use
file://provider.jsorfile://provider.pyfor custom auth, request signing, streaming, retries, multi-step setup, local code, Python agent SDKs, or complex parsing. - Use native model providers for direct model comparisons.
- Use
targetswithinputsfor redteam multi-input systems. Do not invent a singlepromptfield when the real app accepts named inputs.
4. Implement the minimal smoke test
Add or update a config with:
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json- A short
description - Provider or
targetsconfig with{{env.VAR}}for secrets - One or two smoke tests that verify the request reaches the target and the response transform extracts the right field
--no-cacherun commands
When writing a local wrapper, return { output } and include structured errors
when the target response is malformed. JavaScript providers receive config in
constructor options.config and expose callApi(prompt, context); read named
inputs from context.vars. Python providers use file://provider.py or
file://provider.py:function_name; the function takes (prompt, options, context) and reads named inputs from context.get("vars", {}). Set
config.workers: 1 for non-thread-safe SDKs, config.timeout for slow calls,
and config.pythonExecutable/PROMPTFOO_PYTHON for venvs. Add harmless
defaults because validate target may call providers without test-case vars.
5. Validate and run
From the promptfoo repo, use the local build:
npm run local -- validate config -c path/to/promptfooconfig.yaml
npm run local -- validate target -c path/to/promptfooconfig.yaml
npm run local -- eval -c path/to/promptfooconfig.yaml -o output.json --no-cache --no-share
Outside the promptfoo repo, use:
npx promptfoo@latest validate config -c path/to/promptfooconfig.yaml
npx promptfoo@latest validate target -c path/to/promptfooconfig.yaml
npx promptfoo@latest eval -c path/to/promptfooconfig.yaml -o output.json --no-cache --no-share
Inspect the output file for results.stats, response.output, score, and
error; do not rely only on the process exit code.
Use --no-share by default while probing live or internal systems. Remove it
only when the user explicitly wants a cloud share URL.
Common Mistakes
# WRONG: shell-style env vars are literal strings in YAML
apiKey: $API_KEY
# CORRECT: promptfoo renders Nunjucks env references
apiKey: '{{env.API_KEY}}'
# WRONG: flattening a multi-input target into prompt loses attack surface
body:
prompt: '{{prompt}}'
# BETTER: preserve the real app fields
body:
user_id: '{{user_id}}'
message: '{{message}}'
Output Contract
When done, state:
- Connection mode used: live, static, hybrid, or wrapper
- Files created or modified
- Required environment variables
- Safe smoke command with
--no-cache --no-share - What was actually verified, and what remains a TODO
04 同仓库更多 Skills
promptfoo-redteam-setup
Create or refine promptfoo redteam setup configs: purpose, targets, plugins, strategies, frameworks, multi-input target inputs, policy text, grader guidance, contexts, and static-code-derived target/threat mapping. Use when preparing a red team scan plan from live probes, code evidence, or provider configs, or when generating adversarial test cases for QA. Do not use for basic provider wiring alone or for running/evaluating an already-generated redteam scan.
promptfoo-redteam-run
Run, rerun, inspect, and QA promptfoo redteam scans from generated redteam YAML or an existing redteam setup config. Use when executing `promptfoo redteam eval` or `promptfoo redteam run`, exporting results, triaging attack success rate, grader failures, target errors, filter/rerun commands, reports, or CI gates. Do not use for initial provider wiring or for choosing plugins and strategies before generation.
promptfoo-evals
Write, refine, run, and QA non-redteam promptfoo eval suites after the target or provider already works: prompts, vars, test cases, assertions, model-graded rubrics, transforms, datasets, output exports, filters, and CI gates. Use for regression tests and eval-suite authoring. Do not use for connecting a new target/provider, mapping HTTP requests or auth, smoke-testing an endpoint, or redteam plugin/strategy setup; use `promptfoo-provider-setup` for connection work instead.
token-skill
Use this skill when the user explicitly asks to use token-skill and wants the special token.
review-standards
Use this skill when asked to review authentication code for security issues.
review-standards
Use this skill when asked to review authentication code for security issues.