few-shot-learning
Teach models new tasks from a handful of examples — prompting strategies, example selection, and in-context learning mechanics.
- License 许可
- MIT
- 最近更新
- 2026-11-07
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- few-shot-learning
- category
- ai-research
- description
- Teach models new tasks from a handful of examples — prompting strategies, example selection, and in-context learning mechanics.
03 SKILL.md 正文
Overview
Few-shot learning with large language models means teaching a task through examples placed in the prompt — no weight updates. Show the model 2–10 input-output pairs demonstrating the pattern, then give it a new input. The model infers the task from the demonstrations: the output format, the label space, the reasoning style. It's the fastest way to prototype a new capability, and for many tasks it matches or beats fine-tuning, especially when labeled data is scarce.
What makes it work is still partly mysterious, but the practical levers are well mapped: example selection (which demonstrations), example ordering (sequence effects are real), formatting (consistent delimiters the model can latch onto), and the number of shots (more isn't always better — context gets diluted). Modern practice treats the few-shot prompt as a small program to be engineered and evaluated, not as prose to be admired.
The strategic role of few-shot prompting has shifted: it's now the prototyping layer. Prove the task works few-shot, then decide whether to keep the prompt, move to fine-tuning, or distill into a smaller model.
When to use
- Prototyping a new classification, extraction, or transformation task before committing to fine-tuning.
- Tasks with scarce labeled data where fine-tuning would overfit.
- One-off or rapidly-changing tasks where retraining is impractical.
- Steering output format precisely (JSON schemas, specific label sets, stylistic constraints).
- Evaluating whether a task is learnable at all — if few-shot fails badly, the task definition is probably the problem.
- Low-volume tasks where prompt-engineering cost beats fine-tuning cost.
Core concepts
- In-context learning: the model's ability to infer a task from demonstrations without gradient updates. Emerges with scale; larger models need fewer, less carefully chosen examples.
- Example selection: the highest-leverage variable. Strategies: random (surprisingly strong baseline), semantic retrieval (pick training examples most similar to the query via embeddings), diversity sampling (cover the label space and edge cases), and difficulty-based (include hard examples the model gets wrong zero-shot).
- Ordering effects: models are recency-biased — examples near the end of the prompt influence the output more. Also susceptible to majority-label bias (predicting the most frequent label in the demonstrations). Shuffle and test multiple orders.
- Format consistency: use identical delimiters and structure across examples
(
Input: ... Output: ...). The model learns the template as much as the task; inconsistency in the template is inconsistency in the lesson. - Shot count trade-offs: accuracy usually rises steeply to 4–8 shots then plateaus or degrades as the context fills with near-duplicates. Test the curve rather than assuming more is better.
- Calibration: few-shot predictions inherit the label bias of the demonstrations. If your examples are 80% class A, expect inflated class-A predictions. Balance the demonstration set or calibrate outputs.
- Retrieval-augmented selection: for production, retrieve per-query examples from a pool using embeddings. Usually beats a fixed example set, at the cost of a retrieval step.
- Demonstration quality: one wrong example teaches the wrong pattern confidently. Verify every demonstration's correctness — the model trusts them absolutely.
Practical workflow
- Write the zero-shot baseline first. A clear task instruction with no examples. Measure it — this tells you how much the examples are actually adding.
- Curate 8–15 candidate examples. Cover every label/class, include 1–2 tricky edge cases, keep each example short. Quality and coverage beat quantity.
- Verify every example. Each demonstration must be correct and representative. A single wrong example poisons the pattern the model infers.
- Build the prompt template. Fixed instruction + examples in a rigid format + the query in the same format. Delimit examples clearly.
- Select per query (for harder tasks). Retrieve the k most similar examples to each query from your candidate pool using embeddings. This usually beats a fixed example set.
- Evaluate systematically. Test on a held-out set: vary shot count (0, 2, 4, 8), example orders, and selection strategies. Report variance across orders — if results swing wildly, the prompt is fragile.
- Decide: prompt or fine-tune. If few-shot with retrieved examples hits your accuracy bar reliably, ship the prompt. If it's fragile, slow (long contexts cost latency and money), or the task is stable and high-volume, distill the examples into a fine-tune.
Checklist for a production few-shot prompt:
- Evaluated on held-out data with multiple example orders; variance acceptable.
- Label distribution in demonstrations balanced or deliberately chosen.
- Every demonstration verified correct.
- Failure cases reviewed: model fails on genuinely hard inputs, not format confusion.
- Latency and cost of the long prompt acceptable at expected volume.
- Fallback defined for when retrieval finds no good examples.
Common pitfalls
- Testing on the training examples. Evaluating few-shot accuracy on the same examples shown in the prompt measures memorization of the prompt, not learning. Always held-out.
- Ignoring order variance. A prompt that scores 90% with one order and 70% with another is not a 90% prompt. Report the distribution.
- Kitchen-sink prompts. Twenty rambling examples with inconsistent formatting teach the model that the task is "produce rambling text." Fewer, cleaner examples win.
- Label bias blindness. Demonstrations skewed toward one class silently become a prior. Balance or calibrate.
- Assuming more shots = better. Beyond the plateau, extra examples add latency, cost, and sometimes confusion. Find the knee of the curve.
- Fragile formatting. If changing "Output:" to "Answer:" breaks the task, you don't have a robust solution — you have a lucky incantation. Harden the template or fine-tune.
- Unverified demonstrations. Including an example you haven't checked. The model treats demonstrations as ground truth — so must you.
- No per-query retrieval. Using a fixed example set when queries vary widely. Retrieved examples usually win; the retrieval step is cheap compared to the accuracy gain.
04 同仓库更多 Skills
web-scrape
Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.
design-taste-frontend
tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.
test-driven-development
Use when implementing any feature or bugfix, before writing implementation code
systematic-debugging
Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
requesting-code-review
Use when completing tasks, implementing major features, or before merging to verify work meets requirements
brainstorming
You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).