dataset-domain-token-counts-via-metadata-config
Use when a HuggingFace dataset task asks for token counts or filtering by a named domain and the dataset has multiple configs, only one of which contains domain and token-text columns.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-12
- Stars
- ★ 394
- Forks
- 23
01 前言 Front Matter
- name
- dataset-domain-token-counts-via-metadata-config
- flashrsi
- { "level": "topic", "algorithm": "evo_harness", "source_task_ids": [ "count-dataset-tokens" ] }
- description
- Use when a HuggingFace dataset task asks for token counts or filtering by a named domain and the dataset has multiple configs, only one of which contains domain and token-text columns.
03 SKILL.md 正文
Use the metadata config for domain-filtered token counts on OpenThoughts-style datasets
When a dataset task asks for domain-level token counts, do not assume the default config has the needed columns. Inspect the available configs and load the one that actually holds domain labels and reasoning/solution text.
Procedure:
- List configs before loading:
from datasets import get_dataset_config_names print(get_dataset_config_names("ryanmarten/OpenThoughts-1k-sample")) - Read the dataset card or README for documented configs and domain groupings.
- Load the config with the domain and token-text columns, often
metadata, instead ofdefault:from datasets import load_dataset ds = load_dataset("ryanmarten/OpenThoughts-1k-sample", "metadata") - Map README category names to the actual metadata domain values. For example, a
Sciencecategory usually corresponds tophysics,chemistry, andbiology; do not search for a literalsciencedomain value. - Validate before counting:
print(ds["train"].column_names) print(ds["train"].unique("domain"))
Why: on ryanmarten/OpenThoughts-1k-sample the default config only has system and conversations, while domain, deepseek_reasoning, and deepseek_solution live in the metadata config. Counting tokens by domain from default returns no domain labels and can silently count the wrong text.
Supporting evidence: count-dataset-tokens (this batch).
04 同仓库更多 Skills
dynamic-resources
Example skill loaded from resources_discover
hermes-agent
Use the FlashRSI Hermes Snapshot's headless Python runtime, tools, sessions, and compression paths.
skill-installer
Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos).
skill-creator
Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations.
review-agent
Perform a read-only, defect-first review of a specified code change and return every actionable finding. Use when another agent delegates review of uncommitted changes, a base-branch diff, a commit, or custom review instructions.
plugin-creator
Create and scaffold plugin directories for Codex with a required `.codex-plugin/plugin.json`, optional plugin folders/files, valid manifest defaults, and personal-marketplace entries by default. Use when Codex needs to create a new personal plugin, add optional plugin structure, generate or update marketplace entries for plugin ordering and availability metadata, or update an existing local plugin during development with the CLI-driven cachebuster and reinstall flow.