senior-prompt-engineer
Use when the user asks to optimize prompts, design prompt templates, evaluate LLM outputs with an eval set, measure RAG retrieval quality, validate agent/tool configurations, analyze token usage, or design structured-output contracts. Covers eval-driven prompt iteration, RAG metrics (relevance, faithfulness, coverage), agent workflow validation, and token/cost budgeting — all model-agnostic, with three stdlib Python tools.
- License 许可
- MIT
- 最近更新
- 2026-07-04
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- senior-prompt-engineer
- description
- Use when the user asks to optimize prompts, design prompt templates, evaluate LLM outputs with an eval set, measure RAG retrieval quality, validate agent/tool configurations, analyze token usage, or design structured-output contracts. Covers eval-driven prompt iteration, RAG metrics (relevance, faithfulness, coverage), agent workflow validation, and token/cost budgeting — all model-agnostic, with three stdlib Python tools.
02 资源文件 6
- 参考文档references/agentic_system_design.md21.2 KB
- 参考文档references/llm_evaluation_frameworks.md13.0 KB
- 参考文档references/prompt_engineering_patterns.md13.4 KB
- 脚本scripts/agent_orchestrator.py20.9 KB
- 脚本scripts/prompt_optimizer.py18.6 KB
- 脚本scripts/rag_evaluator.py19.9 KB
03 SKILL.md 正文
Senior Prompt Engineer
Eval-driven prompt engineering, RAG quality measurement, and agent workflow validation. Everything here is model-agnostic by design: techniques are framed by what they do, not by which model generation they were observed on, and the tools never hardcode model IDs or pricing — you supply your provider's current rates when you want dollar figures.
Operating Rules
- Never change a prompt without a baseline. Capture metrics first (
--analyze --output baseline.json), then compare every iteration against it. - Eval set before optimization. 10–20 representative cases with expected outputs minimum. If the user has no eval set, build one with them before touching the prompt — optimizing against vibes is the #1 failure mode.
- Prefer platform features over prompt hacks. If the provider offers native structured outputs / JSON schema enforcement, tool-use APIs, or prompt caching, use those instead of "respond ONLY with JSON" incantations. Prompt-level format enforcement is the fallback, not the default.
- Current-generation models need less scaffolding. Don't add chain-of-thought boilerplate, role framing, or few-shot examples reflexively — frontier models often do worse with redundant scaffolding. Add each element only when the eval set shows it helps.
- Cost numbers are always user-supplied. Look up the provider's current per-Mtok pricing and pass it via
--price-per-mtok(never trust a cached price table — including any you remember).
Tools (exact CLIs, all stdlib)
1. Prompt Optimizer — scripts/prompt_optimizer.py
Static analysis: token estimate, clarity/structure scores (0–100), ambiguity + redundancy detection, few-shot example extraction.
# Full analysis (human-readable report)
python3 scripts/prompt_optimizer.py prompt.txt --analyze
# Save machine-readable baseline for later comparison
python3 scripts/prompt_optimizer.py prompt.txt --analyze --json --output baseline.json
# Token estimate; cost only if you supply your provider's current rate
python3 scripts/prompt_optimizer.py prompt.txt --tokens --model claude --price-per-mtok 3.00
# Whitespace/redundancy-trimmed version
python3 scripts/prompt_optimizer.py prompt.txt --optimize --output optimized.txt
# Extract Input/Output few-shot pairs to JSON
python3 scripts/prompt_optimizer.py prompt.txt --extract-examples --output examples.json
# Compare a revision against the saved baseline
python3 scripts/prompt_optimizer.py optimized.txt --analyze --compare baseline.json
--model accepts any string; only the tokenizer family is inferred (names containing "claude" → 3.5 chars/token, otherwise 4.0). Exit 0 on success, 1 on missing file.
2. RAG Evaluator — scripts/rag_evaluator.py
Measures retrieval and grounding quality from two JSON files (formats printed in --help).
python3 scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json
python3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --k 10 --json
python3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --output report.json --verbose
python3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --compare baseline_report.json
Reports context relevance, precision@k, coverage, answer faithfulness, groundedness. Treat relevance < 0.80 as a retrieval problem (chunking/embedding/filtering), not a prompt problem — fix retrieval before rewriting the generation prompt.
3. Agent Orchestrator — scripts/agent_orchestrator.py
Validates agent configs (YAML/JSON): tool wiring, missing required config, loop risk, token estimates.
python3 scripts/agent_orchestrator.py agent.yaml --validate
python3 scripts/agent_orchestrator.py agent.yaml --visualize --format mermaid
python3 scripts/agent_orchestrator.py agent.yaml --estimate-cost --runs 100 \
--input-price-per-mtok 3.00 --output-price-per-mtok 15.00
Without the two price flags, --estimate-cost reports token estimates only. The model: field in the config is informational — any model name is accepted.
Workflows
Prompt Optimization (eval-gated)
- Baseline:
python3 scripts/prompt_optimizer.py current_prompt.txt --analyze --json --output baseline.json - Diagnose from the report: ambiguous verbs ("analyze", "handle"), redundant blocks, missing output contract, token waste.
- Apply one change at a time, in this order of leverage:
Symptom Fix Malformed/unparseable output Native structured outputs / JSON schema if the API supports it; explicit schema-in-prompt otherwise Inconsistent answers across runs Tighten instructions + add 2–3 contrastive examples (one near-miss showing what NOT to do) Misses edge cases Enumerate the edge cases explicitly; add a "when uncertain, do X" rule Token bloat on repeated calls Move stable prefix (system rules, examples) first so prompt caching applies; trim redundancy Wrong reasoning on hard cases Ask for stepwise reasoning in a scratch field the consumer ignores, or use the provider's extended-thinking mode - Re-analyze and compare:
python3 scripts/prompt_optimizer.py revised.txt --analyze --compare baseline.json - Eval gate (must pass before shipping): run the revised prompt over the eval set, write per-case pass/fail to
eval_results.json, then assert:
Pair this structural gate with your task-level eval: the revision must not lose any previously-passing eval case (no-regression rule).python3 scripts/prompt_optimizer.py revised.txt --analyze --json --output revised.json \ && python3 -c " import json, sys r = json.load(open('revised.json')); b = json.load(open('baseline.json')) ok = r['clarity_score'] >= b['clarity_score'] and r['token_count'] <= b['token_count'] * 1.10 sys.exit(0 if ok else 1)" echo "gate exit=$?" # 0 = ship; 1 = regression, iterate again
Few-Shot Example Design
- Define the task contract first (input shape, output shape, edge-case policy).
- Start with zero examples and measure — current models often need none. Add examples only for failure clusters the eval reveals.
- When adding: 3–5 max, ordered simple → edge → negative (what NOT to extract), formatted identically to the real output contract.
- Validate consistency:
python3 scripts/prompt_optimizer.py prompt_with_examples.txt --extract-examples --output examples.jsonand inspect that every extracted pair parses against your schema. - Re-run the eval set; if a case passes only because it resembles an example, add a held-out variant to the eval set.
Structured Output Design
- Write the JSON Schema first (types, enums, required, maxLength).
- Prefer API-native enforcement: structured-outputs / response-schema / tool-call parameters guarantee shape; prompt text cannot.
- Fallback (API without schema support): include the schema rendered as field-by-field rules + one valid example, and instruct "output only the JSON object".
- Gate: pipe 10 eval outputs through a schema validator (
python3 -c "import json,sys; [json.loads(l) for l in sys.stdin]"at minimum); 10/10 must parse, else return to step 2.
RAG Tuning Loop
- Build
questions.json(id, question, reference answer) and capture current retrievals tocontexts.json. python3 scripts/rag_evaluator.py --contexts contexts.json --questions questions.json --output rag_baseline.json- Fix the lowest metric first: relevance → chunking/embeddings/metadata filters; faithfulness → grounding instructions + "answer only from context" + citation requirement; coverage → retrieval k / query expansion.
- Gate:
python3 scripts/rag_evaluator.py --contexts new_contexts.json --questions questions.json --compare rag_baseline.json— every metric must be ≥ baseline; any regression blocks the change.
Agent Config Review
python3 scripts/agent_orchestrator.py agent.yaml --validate— must exit with VALIDATION PASSED; fix every error and warning (missing tool config, unbounded iterations, loop risk).- Check context discipline: each tool description ≤ 1–2 sentences, tool count minimal for the job, stable system prompt placed first (cache-friendly), iteration cap + early-exit condition present.
- Budget:
--estimate-cost --runs Nwith your current prices; if cost/run exceeds budget, cut tools or context before downgrading the model.
References
| File | Contains | Load when user asks about |
|---|---|---|
references/prompt_engineering_patterns.md |
10 prompt patterns with input/output examples | "which pattern?", few-shot design, decomposition, meta-prompting |
references/llm_evaluation_frameworks.md |
Eval metrics, scoring methods, A/B testing | "how to evaluate?", "measure quality", "compare prompts" |
references/agentic_system_design.md |
Agent architectures (ReAct, Plan-Execute, Tool Use) | "build agent", "tool calling", "multi-agent" |
Related Skills
engineering-team/skills/senior-ml-engineer— model deployment and serving (this skill stops at the prompt/eval layer)engineering/rag-architect— RAG system architecture (this skill measures RAG quality; that one designs the pipeline)engineering/agent-designer— full agent system design (this skill validates configs; that one designs the architecture)
04 同仓库更多 Skills
syllabus
Generates a curated supplementary reading list from any course syllabus using Consensus academic search. Grill-me intake (syllabus input format + course audience + year range) plus a grouping forcing-options checkpoint before any search runs — so the reading list matches the course's level and recency need. Parses the syllabus to extract topics and learning outcomes, searches Consensus for recent peer-reviewed papers per topic, and produces a professionally formatted .docx with clickable Consensus links, plain-language summaries calibrated to audience level, and Bloom-higher-order discussion questions tied to course learning goals. Use when the user uploads a syllabus, course outline, or curriculum document and wants supplementary readings (e.g., 'create a reading list from this syllabus', 'find recent papers for my course') — even casual mentions with a syllabus attached should trigger this skill.
research
Default entry point for any research request — a hybrid router that classifies the question deterministically and either delegates to a specialist research skill (pulse for trends/sentiment, grants for NIH funding, litreview for academic literature, syllabus for course reading, patent for prior-art + IP landscape, dossier for entity research) or runs its own plan-decompose-multi-source-search-synthesize-cite fallback workflow when no specialist matches. Always surfaces the routing decision so users can override. Use when the user makes any research request that doesn't obviously match a more-specific specialist skill (e.g., "research [topic]", "look into [topic]", "what do we know about [topic]", "investigate [topic]", "find me information on [topic]", "do some research on [topic]", "I need to understand [topic]"). Output is a markdown briefing (default) or .docx document (on request) with full citations and an audit log.
pulse
Multi-source recency research skill that takes the pulse of any topic across Reddit, Hacker News, the open web, and optionally X/Twitter within a configurable recent window (default 30 days). Forcing intake clarifies topic specificity, angle (trend/sentiment/problems/opportunities/comparison), time window, and platform scope before searching. Returns a synthesized briefing with citations, engagement metrics, and cross-platform pattern analysis. Use when the user requests multi-source recency intelligence on a topic (e.g., 'pulse on [topic]', 'what's happening with [topic]', 'what are people saying about [topic]', 'current conversation about [topic]', 'take the pulse of [topic]', 'trending: [topic]', 'find me info on [topic]'), and for competitor research, trend discovery, tool comparisons, and audience sentiment analysis.
patent
Patent prior-art and landscape intelligence skill — not generic patent help. Commits to one of five sub-use-cases via forcing intake (novelty search / freedom-to-operate / competitive landscape / acquisition diligence / litigation prior-art) before any search runs. Searches Google Patents, Espacenet, USPTO, and optionally Lens.org for citation-graph signals. Output is an editable Word document (.docx) with verdict, ranked closest art (claim-text extracted), CPC-class-aware landscape, family-resolved hits, geographic coverage, FTO flags where applicable, strategy recommendations, and full audit log. Use when the user asks for patent searching or analysis (e.g., 'prior art search for [invention]', 'freedom to operate analysis for [product]'). Produces search signal, not legal advice — always recommends consulting a patent attorney before filing or licensing decisions. Trademark, copyright, and trade-secret questions are out of scope.
notebooklm
Browser automation skill for controlling Google's NotebookLM. Use when the user wants anything done in NotebookLM (e.g., 'open NotebookLM', 'check my [name] notebook', 'ask my notebook about X', 'add [source] to NotebookLM', 'generate a Video Overview from my notebook', 'use NotebookLM Studio'). Handles reading and querying notebooks, adding sources (URLs, text, files, YouTube links, synthesized content), generating Studio outputs (Audio/Video Overviews, Mind Maps, Reports incl. Briefing Doc/Study Guide/FAQ, Flashcards, Quiz, slide decks, infographics — discover the exact set from the live Studio panel; the UI evolves fast), and creating new notebooks. Requires browser automation environment — fails gracefully when unavailable.
litreview
Academic literature orientation skill that searches papers via free keyless APIs (PubMed E-utilities + OpenAlex) by default — with the Consensus MCP as an optional enhancement lane when connected — builds a strategic search plan using PICO (default) or SPIDER / Decomposition / hybrid as fallbacks, and synthesizes findings into a formatted Word (.docx) research guide. Grill-me intake (research question specificity + framework hint + tentative depth) before the recon search; a second forcing checkpoint after Phase 2 confirms framework + sub-areas + depth before searches consume budget. Configurable depth (5/10/20 queries) controls coverage vs. speed. Output is a 'launching pad' — an orientation guide that lets a researcher dive in confidently, not a finished review. Use when the user starts literature-oriented research (e.g., 'litreview on [topic]', 'literature review on [topic]', 'I'm starting a literature review on X', 'I'm writing a paper on X', 'help me research X', 'I'm doing research on X', 'can you help me research X'). Do NOT use for single one-off paper searches wanting a quick list — that's a plain PubMed/OpenAlex (or Consensus) query.