rsi-skill
Evaluate a prompt, system prompt, skill instruction or agent instruction against concrete test cases, then hill-climb it to a measurably better version using any open model. Use when the user wants to test or improve a prompt, asks "is this prompt any good", "make this prompt better", "eval this", "hill-climb", "optimize my system prompt", or has examples of inputs with the answers they expect.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-15
- Stars
- ★ 6
- Forks
- 0
01 前言 Front Matter
- name
- rsi-skill
- license
- Apache-2.0
- description
- Evaluate a prompt, system prompt, skill instruction or agent instruction against concrete test cases, then hill-climb it to a measurably better version using any open model. Use when the user wants to test or improve a prompt, asks "is this prompt any good", "make this prompt better", "eval this", "hill-climb", "optimize my system prompt", or has examples of inputs with the answers they expect.
02 资源文件 1
- 参考文档skill-report.json75.1 KB
03 SKILL.md 正文
Improve a piece of instruction text by measuring it, not by guessing. Score the current
prompt on test cases, look at what fails, have a model propose a revision, keep the
revision only if the score goes up on cases it trained on and does not drop on cases it
never saw, and repeat. Everything runs through scripts/rsi.py (Python 3 standard
library, no install).
It works on any text that steers a model: a system prompt, a classifier instruction, a SKILL.md body, an agent's rules, a few-shot block. It changes instructions, not weights.
Worked example with real numbers: references/example-gsm8k.md (GSM8K, 17% to 92% on held-out problems in four rounds).
1. Pick a model endpoint
Run python3 scripts/rsi.py backend. It uses the first of these that works:
--base-url URL [--api-key-env VAR] [--model M]if the user named an endpoint.- OptiQ Cloud when
OPTIQ_CLOUD_KEYis set (https://cloud.mlx-optiq.com/api/v1, open models). - A local
optiq serveon127.0.0.1:8080(open models on an Apple Silicon Mac; nothing leaves the machine). - Any OpenAI-compatible API via
OPENAI_BASE_URLandOPENAI_API_KEY.
If none is available, the command prints how to set one up. Ask the user which they prefer; never invent keys. The test cases and the prompt are sent to the chosen endpoint, so confirm with the user before using a remote one for private data.
Pass --model when the endpoint serves several models. For reasoning models:
--reasoning-effort none|low|high where the API supports it, --no-think for servers that
honour enable_thinking, or a larger --max-tokens. Empty answers are reported as such.
2. Write the task and the cases
python3 scripts/rsi.py init DIR creates DIR/prompt.md and DIR/cases.jsonl. Put the
user's current prompt in prompt.md. Each line of cases.jsonl is one case:
{"input": "text the model receives", "expect": "what a good answer has", "check": "contains"}
| check | passes when |
|---|---|
exact |
the answer equals expect (case, quotes and trailing punctuation ignored) |
contains |
the answer contains expect (case-insensitive) |
regex |
the answer matches the regex in expect |
number |
the last number in the answer equals expect (commas, $, % ignored) |
json:<field> |
a JSON object in the answer has field equal to expect |
judge |
the same model grades the answer PASS against the rubric in expect |
Prefer the cheapest check that captures "correct"; use judge only for open-ended
answers. Aim for 10 to 40 cases that cover the real spread, including the hard ones the
user has seen fail. With fewer than 6 there is no held-out set, so say the result may
overfit. Help the user write cases from real inputs rather than inventing them all.
3. Measure, then climb
python3 scripts/rsi.py eval DIRscores the current prompt and lists every failure. Show the user the baseline before changing anything.python3 scripts/rsi.py run DIR --rounds 5holds 30% of the cases aside, then each round asks the model to fix the failures and keeps a revision only if training cases improve and held-out cases do not get worse. Revisions that quote test inputs verbatim are rejected as overfitting. It stops early when every training case passes.--propose-model Muses a stronger model on the same endpoint to write revisions.- Results land in
DIR/results/:best_prompt.mdandhistory.json(every round's scores and candidate prompt).
4. Report
Tell the user the baseline and final scores (train and held out), the model and endpoint,
and show the diff between the original and the best prompt. Name any case that still
fails and why. Apply best_prompt.md only after the user agrees.
Where to try it next
Agent-style tasks with checkable answers make good targets. The LocalLLaMA mini benchmarks on Hugging Face are small enough to iterate on: typed-decisions, typed-extraction, browsecomp-plus-mini, terminal-bench-mini and deepswe-mini. Ask before downloading a dataset.
Notes
- A higher score means the prompt fits these cases better; it is only as good as the cases.
- One run is one sample. Re-run
evalon the best prompt to confirm before trusting it.
04 同仓库更多 Skills
serpmaven
Scrape Google search results (SERP) to JSON with SERP Maven (serpmaven) - a real Chrome browser on a residential proxy with sticky sessions. Run it as a CLI, a Python library, a local HTTP search API (`serve`), an MCP server or a Docker image; set the proxy login, workers, pages and cost tracking. Use when the user wants Google results, rank tracking, a self-hosted SERP API, or live Google search for an AI agent.
revnu-landing-page-audit
Find out why a landing page isn't turning visitors into signups, demos or sales. Audits one page against what the ad, post or search that sends people there promised, then checks the first screen, proof, call to action, friction and mobile. Returns at most 7 prioritized fixes, plus what is fine and what could not be checked; rewrites specific lines and fixes the page in the repo when asked. Never promises a conversion lift. Works without a Revnu account. Use when the user says or asks: "audit my landing page", "why isn't this page converting", "review my homepage", "my ads get clicks but no signups", "people bounce from the pricing page", "CRO review", "improve conversion rate", "is my hero section clear", "roast my landing page", "rewrite my headline", "what's wrong with this page", "landing page feedback", "make the signup page convert better". Not for SEO or AI-search audits, page speed engineering on its own, or building a new page from scratch.
revnu-growth-context
Write down once who buys, what they're stuck on, what it costs, what proof you have and what you've already tried, so every growth task starts from facts instead of a questionnaire. Creates or updates .agents/revnu-growth.md from the repo, the live site and the user, and imports an existing product-marketing.md. Use when the user says "set up our marketing context", "who is our ideal customer", "write our ICP", "describe the product for marketing", "positioning doc", "what's our brand voice", "remember this about our customers", "log what we tried", "what have we learned from marketing", "we changed pricing, update the context", "update our growth notes", "summarize the business for the agent", "import my product-marketing.md", "start a growth doc", or before a first marketing, SEO, outreach or launch task when no context file exists. Works without a Revnu account.
revnu-get-listed
Get onto the third-party pages AI answers and search cite when buyers ask for the best tool in your category: "best X" roundups, directories, review sites, GitHub awesome lists, community threads and comparison pages. Finds which ones are cited, ranks them by fit and effort, and drafts each submission, PR or pitch by the target's own rules. Use when the user says "get us listed", "which directories should we submit to", "how do we get into best-of lists", "why does ChatGPT recommend competitors and not us", "get mentioned in AI answers", "submit to awesome lists", "get on G2", "where are our competitors listed", "pitch listicle authors", "find backlink opportunities", "launch directories", "get into roundups", "Product Hunt alternatives", "where should we be listed", "get cited by Perplexity". Never submits, pays or posts without a yes. Works without a Revnu account.
revnu-ai-readiness-audit
Audit whether your website gives ChatGPT, Perplexity, Claude, Gemini and Google's AI Overviews something worth quoting, and fix the pages that don't. Use when the user says or asks "is my site ready for AI search", "audit my site for AI answers", "AEO audit", "GEO audit", "why doesn't ChatGPT quote our pricing", "make our pages citable by AI", "do I need llms.txt", "does schema help with ChatGPT", "check our structured data", "is our content visible without JavaScript", "AI SEO audit", "optimize our homepage for AI answers", "LLM SEO check", "why does AI get our product wrong", "fix our site for AI Overviews". Checks fetchability and indexability, answers in the raw HTML, entity facts, schema (including JS-injected markup), headings, dates and llms.txt, then gives at most 10 prioritised fixes and edits the repo when asked. Works without a Revnu account.
proxy-troubleshooting
Diagnose residential proxy failures, with NodeMaven's gateway as the reference - 407 with correct credentials, 406, 410, 500, a connection that hangs, ERR_TUNNEL_CONNECTION_FAILED in a browser, Playwright or Puppeteer ignoring proxy credentials, a sticky session that changes IP, geo-targeting that is silently ignored, passwords with special characters, and unexpected traffic from headless Chrome. Use when a proxy connection fails, returns the wrong location, or costs more traffic than expected.