dataset-curation
Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.
- License 许可
- MIT
- 最近更新
- 2026-11-10
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- dataset-curation
- description
- Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.
02 资源文件 2
- 参考文档references/formats-and-templates.md6.5 KB
- 参考文档references/synthetic-data.md12.6 KB
03 SKILL.md 正文
Dataset Curation
This skill assumes finetuning-method-selection
already routed here — the next step is preparing
data, not choosing a method. What follows: format
selection by target method, the template/packing
mechanics behind the most common silent training
failures, rules for mixing in synthetic data
without collapse, and the dataset card that closes
out Phase 2 before a run starts.
Input: raw examples (demonstrations, preference
judgments, or task prompts) plus a routing decision
from finetuning-method-selection.
Output format: a formatted, packed, validated
JSONL dataset plus a completed dataset card — the
Phase 2 artifact /finetune checks before launching
training.
Format Selection
| Method | Shape | Rows |
|---|---|---|
| SFT, single-turn | Instruct (instruction/response or prompt/completion) |
~1,000+ floor |
| SFT, multi-turn | Conversation / ChatML messages list |
~1,000+ floor |
| DPO / ORPO | Preference pair (prompt, chosen, rejected) |
Method-dependent, see preference-optimization |
| KTO | Unpaired (prompt, completion, label) |
Method-dependent, see preference-optimization |
| GRPO / RLVR | Prompt-only (prompt + verifier metadata) |
Method-dependent, see grpo-rlvr-training |
-
~1,000+ rows is the recommended floor for SFT, not a target. Below it, a handful of low-quality or duplicate examples can dominate the gradient; above it, quality over quantity — a smaller verified, deduplicated set beats a larger noisy one.
-
The ChatML shape, for orientation; the other four formats plus a ShareGPT conversion note live in
references/formats-and-templates.md:{"messages": [ {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."} ]}
Chat Templates and Loss Masking
Apply the target model's chat template before any concatenation or packing, never after — packing raw text and templating the packed blob afterward corrupts turn boundaries, landing role markers in the wrong place relative to each example.
-
Train on assistant responses only. Mask the loss (
-100in the labels tensor) over system/user turns and the template's own role markers — only assistant-turn content tokens contribute to loss. -
Template/tokenizer mismatches are a top silent failure mode. A model trained against one chat template but served or evaluated with a different one degrades without erroring. Verify the same template string used in training is applied at inference and eval time.
-
Keep the dataset in
messagesshape and let the trainer template and mask it (assistant_only_loss=Truein current TRL) — pre-rendering to a flat text field destroys the turn boundaries masking needs. Full code sketch:references/formats-and-templates.md. Sanity-check before training — decode only unmasked positions; expect only assistant text:keep = batch["labels"][0] != -100 print(tokenizer.decode(batch["input_ids"][0][keep]))
Packing
Without packing, compute goes to padding —
variable-length examples batched at a fixed
sequence length waste the gap between each
example's length and the batch's max. How much
needs measurement: compare the sum of example
token counts with batches × max_length on your
data. Packing concatenates multiple examples into
one sequence up to the max length, cutting most of
that waste. Padding-free (position-id) packing
keeps examples from attending to each other
without a block-diagonal mask; prefer it where the
trainer supports it.
-
Packing changes batch semantics. A packed sequence can contain several original examples, so "steps per epoch" and any LR schedule keyed to example count shift once packing is on — recompute schedule milestones against packed-sequence count.
-
MANDATORY: decode and manually inspect 5–10 packed sequences before scaling to a full run. Confirm example boundaries land where expected, template markers are intact per sub-example, and the loss mask is still assistant-only within each packed sequence. Not optional — packing bugs are silent (the loss curve looks normal) and only surface in eval quality, hours later:
for seq in packed_dataset.select(range(10)): print(tokenizer.decode(seq["input_ids"]))
Synthetic Data Rules
- Keep ≥25% real data as a collapse guard.
Training on a growing share of model-generated
data without a real-data floor drives measurable
quality collapse over successive generations —
25% real is the minimum that holds the line.
General-domain replay rows
count toward this floor —
"real" means "not generated
for this task from this
student," not "human-authored."
An all-synthetic-by-construction
dataset can meet the ≥25% floor
through replay alone (see
references/synthetic-data.md's Replay-Mix Construction recipe); state which rows count as "real" in the dataset card rather than leaving the floor structurally unmeetable. - Magpie and rejection sampling are the workhorses. Magpie extracts prompts from the model's own template prior; rejection sampling generates several candidates per prompt and keeps only the ones a filter passes. Both beat naive single-shot generation.
- Targeted, student-aware generation usually beats static generation — aiming at the student's actual failure modes hits a quality bar with fewer filtered examples. The gain needs measurement on your task.
- Measure the accept rate on a pilot batch and plan volume from it: raw generations needed ≈ target rows ÷ accept rate.
- Generation-method ranking, filter funnel, replay-
mix construction, and distillation pattern:
references/synthetic-data.md.
The Dataset Card
Every dataset that reaches training gets a card —
the required Phase 2 artifact /finetune checks
before launching. The card is not free-form
documentation; it MUST carry these fields:
- Provenance — where every row came from (real
source(s), synthetic method(s), or both),
traceable to
trace-to-training-dataoutput. - Counts — total rows, and rows per split (train/eval/held-out) if split.
- Synthetic/real ratio — the measured ratio, checked against the ≥25% real floor above.
- Dedup method — exact-match, semantic
(embedding threshold), or both; see the filter
funnel in
references/synthetic-data.md. - Template used — the exact chat template
string/identifier, kept consistent through
inference and eval — this is what ties an
eval-harness-firstrun back to the checkpoint. - Packing config — whether packing was used, max sequence length, and confirmation the 5–10-sequence manual inspection above was done.
A dataset missing any of these six fields isn't
ready for /finetune — the card is a gate, not a
summary written after the fact.
Phase 2 Exit Checklist
Before handing off to /finetune, confirm:
- Format matches the method (table above).
- Template applied before concatenation.
- Loss masked to assistant turns only.
- 5–10 packed sequences decoded and read.
- ≥25% real data in the final mix.
- Dataset card complete — all six fields.
References
references/formats-and-templates.md— JSONL examples per format, current-TRL masking code, and the ShareGPT conversion note.references/synthetic-data.md— generation-method ranking, filter funnel, replay-mix construction, and teacher→student distillation pattern.
Related skills: finetuning-method-selection routes
here; lora-qlora-recipes, vision-sft, and
preference-optimization consume the datasets this
skill produces; trace-to-training-data is the
provenance source for graded-trajectory datasets;
eval-harness-first grades the resulting checkpoint.
04 同仓库更多 Skills
no-description
暂无简介
good-skill
Fixture skill that satisfies every rule.
broken-link
Links to a reference file that does not exist.
Bad_Name
Name violates the kebab-case rule and does not match the directory.
skill-authoring
Write, review, and validate portable agent skills in SKILL.md format. Use when creating a new skill, editing an existing one, or asked whether a skill follows the Agent Skills spec.
weigh-bags
Record scale readings for each bag — empty (tare), outbound, current, final — and reconcile them against the itemised inventory to find unaccounted weight. Use when the user gives a bag weight, says how heavy something is, weighs their luggage, or asks how close to the limit they are. Appends to bags.yaml and runs scripts/reconcile.py.