[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71FN6EG0E0TG5HVC5ZGV6PW":3,"skill-same:01M71FN6EG0E0TG5HVC5ZGV6PW":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":49,"repositoryHtmlUrl":50,"repositoryHomepageUrl":6,"ownerHtmlUrl":51,"ownerBlogUrl":52,"ownerDescription":6,"updatedAt":53,"install":54,"resources":67},"01M71FN6EG0E0TG5HVC5ZGV6PW","create-task",null,"Create a new Harbor task for evaluating agents. Use when the user wants to scaffold, build, or design a new task, benchmark problem, or eval. Guides through instruction writing, environment setup, verifier design (pytest vs Reward Kit vs custom), and solution scripting.","pku-liang\u002Fhwe-bench","pku-liang","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F81861363?v=4","01M71FMJQ2VZ4YEERAS5FJ438T","PKU Yun (Eric) Liang Research Group","01M71FMJR6AXWMX6K35KS4ZS6G",73,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fpku-liang\u002Fhwe-bench\u002Fblob\u002Fmain\u002Fdeps\u002Fharbor\u002Fskills\u002Fcreate-task\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"task","create task","create","use","custom","verifier","environment","for","pytest","org","benchmark","guides","when","writing","new","kit","design","eval","new task","problem","2026-11-10 10:12:30",false,"---\nname: create-task\ndescription: Create a new Harbor task for evaluating agents. Use when the user wants to \n  scaffold, build, or design a new task, benchmark problem, or eval. Guides through \n  instruction writing, environment setup, verifier design (pytest vs Reward Kit vs \n  custom), and solution scripting.\nargument-hint: [org\u002Ftask-name]\n---\n\nGuide the user through creating a new Harbor task end-to-end. Don't just dump commands — \nwalk them through each decision, especially around the verifier (which is usually the \nhardest part).\n\n## Step 1: Scaffold the task\n\n```bash\nharbor task init \"\u003Corg>\u002F\u003Ctask-name>\"\n```\n\nUseful flags:\n- `--description \"...\"`\n- `--author \"Jane Doe \u003Cjane@example.com>\"` (repeat for multiple authors)\n- `--no-pytest` — skip the pytest test template (use if planning Reward Kit or custom verifier)\n- `--no-solution` — skip solution\u002F directory\n- `--metadata-template path.toml` — pre-populate task.toml\n\nProduces:\n```\n\u003Ctask-name>\u002F\n├── instruction.md         # Task prompt for the agent\n├── task.toml              # Config and metadata\n├── environment\u002FDockerfile # Container definition\n├── solution\u002Fsolve.sh      # Reference solution (optional)\n└── tests\u002Ftest.sh          # Verifier script\n```\n\nIf the user wants a **multi-step task** (ordered steps with per-step\ninstructions, tests, and early stopping against a shared container), scaffold\nthe single-step layout first, then convert to the `steps\u002F` layout described in\nthe *Multi-step tasks* section below.\n\n## Step 2: Write instruction.md\n\nThis is the prompt the agent receives. Help the user write it clearly:\n\n- **State the goal concretely** — what file to create, what behavior to produce\n- **Specify expected outputs** — paths, formats, content\n- **Include constraints** — language, tools, approach\n- **Don't leak the tests** — describe what \"done\" looks like, not how you'll check it\n\nExample (from the ssh-key-pair tutorial):\n```markdown\n# SSH Key Pair Generation\n\nGenerate an SSH key pair in the files `~\u002F.ssh\u002Fid_rsa` and `~\u002F.ssh\u002Fid_rsa.pub`.\n\nDon't make them password protected.\n```\n\n## Step 3: Build the environment\n\nEdit `environment\u002FDockerfile` to install dependencies the task needs. The agent works \ninside this container.\n\n```dockerfile\nFROM ubuntu:24.04\nWORKDIR \u002Fapp\n\n# Install what the task requires — NOT the solution\nRUN apt-get update && apt-get install -y openssh-client && rm -rf \u002Fvar\u002Flib\u002Fapt\u002Flists\u002F*\n```\n\nFor multi-container setups, use `environment\u002Fdocker-compose.yaml` instead (note: most \ncloud sandbox providers only support Dockerfile).\n\n**Test the environment interactively** before writing the solution or tests:\n```bash\nharbor task start-env -p \"\u003Ctask-path>\" -e docker -a -i\n```\n\nThis is usually where task authors realize something is missing from the Dockerfile.\n\n## Step 4: Decide how to verify\n\n**This is the most important decision.** Ask the user: *\"How do you want to grade this \ntask?\"* Then help them pick:\n\nAlso ask: *\"Should the verifier run in the same environment as the agent, or in a\nseparate verifier environment?\"*\n\n- Use the default shared environment when tests need to inspect the agent's full\n  workspace, installed tools, or services.\n- Use a separate verifier environment when grading code, dependencies, API keys,\n  or OS requirements should stay hidden from the agent, or when verification\n  should run from a clean image.\n\nFor a separate verifier container, `tests\u002F` is the verifier image build context\nand the image must provide `\u002Ftests\u002Ftest.sh` (Linux) or `\u002Ftests\u002Ftest.bat`\n(Windows). Harbor copies `\u002Flogs\u002Fartifacts` and configured artifacts into the\nverifier environment, not the agent's whole workspace.\n\n```toml\n[verifier]\nenvironment_mode = \"separate\"\n\n[verifier.environment]\ndocker_image = \"ubuntu:24.04\"\n```\n\n### Option A: Reward Kit (recommended for most cases)\n\nUse when the verifier has multiple criteria, needs partial credit, uses an LLM\u002Fagent \njudge, or would benefit from composable reusable checks. See the `rewardkit` skill.\n\nGood fit signals:\n- Multiple things to check (file exists + content correct + command works)\n- Subjective quality dimensions (readability, correctness of prose)\n- Want partial credit rather than pass\u002Ffail\n- Want to compose built-ins like `file_contains`, `command_succeeds`, `json_key_equals`\n\n`tests\u002Ftest.sh`:\n```bash\n#!\u002Fbin\u002Fbash\nuvx --from 'harbor-rewardkit==0.1.*' rewardkit \u002Ftests\n```\n\nNote: the package is named `harbor-rewardkit` but the executable is `rewardkit`,\nhence `--from 'harbor-rewardkit==0.1.*' rewardkit`. Running\n`uvx harbor-rewardkit` directly will fail.\n\nThen add `tests\u002Fchecks.py` and\u002For `tests\u002Fjudge.toml`. Invoke the `rewardkit` skill to \ndesign the criteria.\n\n### Option B: pytest (good for deterministic unit-style checks)\n\nUse when the verification is straightforward assertion-style Python. Default template if \n`--no-pytest` wasn't passed.\n\n`tests\u002Ftest.sh`:\n```bash\n#!\u002Fbin\u002Fbash\napt-get update && apt-get install -y curl\ncurl -LsSf https:\u002F\u002Fastral.sh\u002Fuv\u002F0.9.7\u002Finstall.sh | sh\nsource $HOME\u002F.local\u002Fbin\u002Fenv\n\nuvx --with pytest==8.4.1 pytest \u002Ftests\u002Ftest_outputs.py\n\nif [ $? -eq 0 ]; then\n  echo 1 > \u002Flogs\u002Fverifier\u002Freward.txt\nelse\n  echo 0 > \u002Flogs\u002Fverifier\u002Freward.txt\nfi\n```\n\nExample `tests\u002Ftest_outputs.py`:\n```python\nfrom pathlib import Path\n\ndef test_file_exists():\n    assert (Path.home() \u002F \".ssh\" \u002F \"id_rsa\").exists()\n```\n\n### Option C: Custom shell\n\nFor simple single-command checks (e.g. a binary pass\u002Ffail from one command):\n```bash\n#!\u002Fbin\u002Fbash\nif diff -q \u002Fapp\u002Foutput.txt \u002Ftests\u002Fexpected.txt; then\n  echo 1 > \u002Flogs\u002Fverifier\u002Freward.txt\nelse\n  echo 0 > \u002Flogs\u002Fverifier\u002Freward.txt\nfi\n```\n\n### Reward file format (all options)\n\n- `\u002Flogs\u002Fverifier\u002Freward.txt` — single number (usually `0` or `1`)\n- `\u002Flogs\u002Fverifier\u002Freward.json` — `{\"accuracy\": 0.95, \"runtime_sec\": 1.2}` for multiple metrics\n\n**Always use absolute paths in `test.sh`.**\n\n## Step 5: Write the solution\n\nWrite `solution\u002Fsolve.sh` — a script that actually solves the task. The Oracle agent runs \nthis to sanity-check that the task is solvable and the tests pass on a correct solution.\n\n```bash\n#!\u002Fbin\u002Fbash\nssh-keygen -t rsa -f ~\u002F.ssh\u002Fid_rsa -N \"\"\n```\n\nMake it executable: `chmod +x solution\u002Fsolve.sh`.\n\n## Step 6: Configure task.toml\n\nWalk through the important fields:\n\n```toml\n[task]\nname = \"\u003Corg>\u002F\u003Ctask-name>\"\nversion = \"1.0.0\"\ndescription = \"One-line description\"\nkeywords = [\"jax\", \"mnist\", \"rewardkit\"]  # always populate — used for search\u002Ffiltering\n\n[metadata]\ndifficulty = \"easy\" | \"medium\" | \"hard\"\ncategory = \"programming\" | \"machine-learning\" | \"gpu\" | ...\ntags = [\"...\"]\n\n[agent]\ntimeout_sec = 120.0       # How long the agent has\n\n[verifier]\ntimeout_sec = 600.0       # How long tests have\n\n[environment]\nnetwork_mode = \"public\"   # Baseline at env start (defaults to public)\ncpus = 1                  # CPU cores\nmemory_mb = 2048          # RAM in MB\nstorage_mb = 10240        # Disk in MB\n```\n\n**Always populate `keywords`.** Pick 3–8 lowercase tokens covering the domain\n(language\u002Fframework\u002Fbenchmark family), the verifier style (`rewardkit`,\n`judge-grading`, `pytest`), and any notable hardware (`gpu`). They're surfaced in\n`harbor datasets list` and registry search.\n\n### Network policy\n\nNetwork access has three layers:\n\n1. **Baselines** — set when an environment starts, restored between phases\n2. **Phase overrides** — optional; only during `agent.run()` or `verify()`\n3. **Run-time merges** — `--allow-environment-host`, `--allow-agent-host` on `harbor run`\n\n| Field | Layer | When applied |\n| --- | --- | --- |\n| `[environment].network_mode` | Baseline | Agent env start; shared verifier uses this too |\n| `[verifier.environment].network_mode` | Baseline | Separate verifier env start |\n| `[agent].network_mode`, `[steps.agent].network_mode` | Override | During matching `agent.run()` |\n| `[verifier].network_mode`, `[steps.verifier].network_mode` | Override | During matching `verify()` |\n| `--allow-environment-host` | Run-time | Merged into `environment.extra_allowed_hosts` → `[environment]` baseline |\n| `--allow-agent-host` | Run-time | Merged into `agent.extra_allowed_hosts` → agent phase allowlist |\n\nModes: `public`, `no-network`, or `allowlist` with `allowed_hosts = [\"pypi.org\"]`\n(exact hostnames, IPv4\u002FIPv6 address literals or CIDR ranges, or leading wildcard hostnames, when supported by the selected environment; not URLs,\nports, or paths). Omitting `[environment].network_mode` defaults to `public`.\n\n`[agent]` \u002F `[verifier]` are **optional phase overrides** — only applied when set\n**and** different from the phase baseline. Matching the baseline is a no-op.\n\n**Shared verifier** (default): verifier runs in the agent container; baseline is\n`[environment]`. **Separate verifier**: baseline is `[verifier.environment]` if\nset, else a copy of `[environment]`.\n\n```toml\n# Agent starts offline; agent phase opens network; verifier stays offline\n[environment]\nnetwork_mode = \"no-network\"\n\n[agent]\nnetwork_mode = \"public\"\n\n[verifier]\nnetwork_mode = \"no-network\"\n```\n\nIf a phase override differs from its baseline, the environment provider must\nsupport `dynamic_network_policy` (E2B does; plain Docker does not). Prefer\n`environment_mode = \"separate\"` when agent and verifier need different baselines\nwithout runtime switching:\n\n```toml\n[environment]\nnetwork_mode = \"no-network\"\n\n[verifier]\nenvironment_mode = \"separate\"\n\n[verifier.environment]\nnetwork_mode = \"public\"   # Verifier baseline — not a phase override\n```\n\nRun-time host flags for eval jobs without editing `task.toml`:\n\n```bash\nharbor run -p \"\u003Ctask-path>\" -a oracle \\\n  --allow-environment-host pypi.org \\   # env start (Dockerfile apt\u002Fcurl)\n  --allow-agent-host files.pythonhosted.org   # agent.run() only\n```\n\nOn a `public` baseline, run-time host flags emit a warning and are ignored.\n\nExamples: `examples\u002Ftasks\u002Fnetwork-policy-matrix\u002F`. Full reference:\n`docs\u002Fcontent\u002Fdocs\u002Ftasks\u002Findex.mdx` (Network policy section).\n\nFor Reward Kit judges needing API keys:\n```toml\n[verifier.env]\nANTHROPIC_API_KEY = \"${ANTHROPIC_API_KEY}\"\n```\n\n## Step 7: Verify with the Oracle agent\n\n```bash\nharbor run -p \"\u003Ctask-path>\" -a oracle\n```\n\nOracle runs `solution\u002Fsolve.sh` and then the verifier. Reward should be `1.0`. If it's \nnot, debug in this order:\n1. Does `solve.sh` actually solve it? (`start-env -a -i` and run it manually)\n2. Does the verifier correctly detect success? (check `\u002Flogs\u002Fverifier\u002F` output)\n3. Are paths correct? (absolute vs relative)\n4. Are dependencies installed in the Dockerfile?\n\n## Step 8: Test with a real agent (optional)\n\n```bash\nharbor run -p \"\u003Ctask-path>\" -a terminus-2 -m anthropic\u002Fclaude-sonnet-4-6\n```\n\nIf the task is too easy (every model 1.0) or impossible (every model 0.0), consider \nadjusting difficulty.\n\n## Step 9: Update README.md (always the final step)\n\n`harbor task init` leaves `README.md` as a stub. Before wrapping up, populate it so\nfuture humans (and agents) can understand the task without reading every file.\nInclude:\n\n- **What the agent does** — one paragraph, link to `instruction.md`.\n- **Environment** — base image, key installed packages, cached data, hardware\n  (GPU\u002FCPU\u002FRAM), agent timeout.\n- **Verifier** — for Reward Kit tasks, a table of reward dimensions with type\n  (programmatic \u002F LLM judge \u002F agent judge) and what each measures; how they're\n  aggregated.\n- **Layout** — a tree of the task directory with one-line annotations.\n- **Running** — the concrete `harbor run` commands (Oracle + real agent), with the\n  right provider flag if the task needs a GPU.\n\nTreat this as docs, not marketing — the reader wants to know *what they'd need to\nchange* to modify the task.\n\n## Multi-step tasks\n\nUse when the work splits into ordered phases that should be scored separately,\nwhen you want early stopping between phases, or when you're testing an agent's\nability to build on its own prior work. Steps share one container; files\npersist across steps.\n\n### Directory layout\n\nReplace the task-root `instruction.md`, `tests\u002F`, and `solution\u002F` with a\n`steps\u002F` directory containing one sub-directory per step:\n\n```\n\u003Ctask-name>\u002F\n├── task.toml\n├── environment\u002FDockerfile       # Built once, shared across all steps\n├── steps\u002F\n│   ├── scaffold\u002F\n│   │   ├── instruction.md       # Prompt for this step\n│   │   ├── workdir\u002F             # Uploaded to WORKDIR before the agent runs\n│   │   │   └── setup.sh         # Optional pre-agent hook (reserved filename)\n│   │   ├── tests\u002Ftest.sh        # Per-step verifier\n│   │   └── solution\u002Fsolve.sh    # Per-step Oracle solution (optional)\n│   ├── implement\u002F\n│   │   └── ...\n│   └── document\u002F\n│       └── ...\n└── tests\u002F                       # Optional shared helpers + fallback test.sh\n```\n\nTask-level `tests\u002F` is uploaded to `\u002Ftests` for each step's verification, then\nthe step's own `tests\u002F` is layered on top (same-name files win). Use this for\nshared helpers.\n\n`steps\u002F{name}\u002Fworkdir\u002Fsetup.sh` is a **reserved filename**: if present, it runs\nafter the `workdir\u002F` upload and before the agent, as the step's agent user,\nwith cwd = WORKDIR. Non-zero exit aborts the step and the trial. Have it\n`rm -- \"$0\"` on its last line if the agent shouldn't see it.\n\n### task.toml\n\n```toml\nschema_version = \"1.4\"\n\n[task]\nname = \"\u003Corg>\u002F\u003Ctask-name>\"\nversion = \"1.0.0\"\n\n# How per-step rewards roll up into the trial-level verifier_result.\n# \"mean\" (default): per-key mean across steps that produced a result.\n# \"final\": the last step's verifier_result verbatim.\nmulti_step_reward_strategy = \"mean\"\n\n[[steps]]\nname = \"scaffold\"              # Must match the directory under steps\u002F\nmin_reward = 1.0               # Abort trial if this step's reward \u003C 1.0\n[steps.agent]\ntimeout_sec = 60.0             # Overrides task-level [agent].timeout_sec\n[steps.verifier]\ntimeout_sec = 30.0\n\n[[steps]]\nname = \"implement\"\n# Dict form gates on specific keys from a multi-dim reward:\nmin_reward = { correctness = 0.8, style = 0.5 }\n[steps.agent]\ntimeout_sec = 120.0\n[steps.verifier]\ntimeout_sec = 30.0\n\n[[steps]]\nname = \"document\"\n[steps.agent]\ntimeout_sec = 60.0\n[steps.verifier]\ntimeout_sec = 30.0\n```\n\nPer-step overrides available: `agent.timeout_sec`, `agent.user`,\n`agent.network_mode`, `verifier.timeout_sec`, `verifier.env`, `verifier.user`,\n`verifier.network_mode`, `verifier.environment_mode`, `verifier.environment`,\n`steps.verifier.environment.network_mode`, `healthcheck.*`, `artifacts`. Unset\nfields fall back to the task-level values.\n\n### Choosing a reward strategy\n\n- **`\"mean\"`** — aggregate signal across all steps; good for continuous\n  progress rewards.\n- **`\"final\"`** — last step's verifier_result is the trial reward. Right when\n  the final step is an end-to-end check whose dict already represents the full\n  task. Caveat: if `min_reward` triggers an early abort, `\"final\"` uses the\n  *aborted* step's result, not the intended final step.\n\n### Artifacts\n\nStep-level `artifacts` are collected into `steps\u002F{name}\u002Fartifacts\u002F` after that\nstep's verification. Task-level and trial-level artifacts are collected at\nevery step in addition to the step-level ones.\n\n### Oracle verification\n\n`harbor run -p \"\u003Ctask-path>\" -a oracle` runs each step's `solution\u002Fsolve.sh`,\nthen each step's verifier, in order. Trial reward should be `1.0` across the\naggregation strategy.\n\n### Full reference + worked example\n\n- Docs: `docs\u002Fcontent\u002Fdocs\u002Ftasks\u002Fmulti-step.mdx`\n- Example task: `examples\u002Ftasks\u002Fhello-multi-step-advanced\u002F`\n\n## Special features (mention if relevant)\n\n- **Network policy**: Baselines on `[environment]` \u002F `[verifier.environment]`; phase\n  overrides on `[agent]` \u002F `[verifier]`; see *Network policy* under Step 6\n- **MCP servers**: Add `[[environment.mcp_servers]]` in task.toml for agent tooling\n- **Healthcheck**: Add `[environment.healthcheck]` for services that need to be ready\n- **GPU**: Set `environment.gpus` and optionally `environment.gpu_types`\n- **Pre-built image**: Set `environment.docker_image` instead of building from Dockerfile. You can omit `environment\u002FDockerfile` and place runtime files (configs, scripts, data) directly under `environment\u002F`; Harbor uploads them into the container workdir when the environment starts.\n- **Non-root user**: Set `agent.user` \u002F `verifier.user` for isolation\n\n## Common pitfalls\n\n- Forgetting to write the reward file → task \"passes\" silently with reward 0\n- Using relative paths in `test.sh` → breaks when Harbor runs it from a different cwd\n- Installing the solution into the Dockerfile → agent already gets the answer\n- Test script leaks into `instruction.md` → agent sees the rubric and gaming becomes trivial\n- Forgetting `chmod +x solution\u002Fsolve.sh` → Oracle agent fails\n- Leaving `keywords = []` in task.toml → task is invisible to registry search\n- Leaving `README.md` as a stub → teammates have no way to understand the task at a glance\n- Putting `network_mode` on `[agent]` expecting it to apply at env start → use\n  `[environment].network_mode` for the baseline; agent\u002Fverifier fields are phase overrides\n- Phase override differs from baseline on Docker → task rejected unless provider supports\n  `dynamic_network_policy`; use separate verifier env or match the baseline instead\n","Guide the user through creating a new Harbor task end-to-end. Don't just dump commands — \nwalk them through each decision, especially around the verifier (which is usually the \nhardest part).\n\n## Step 1: Scaffold the task\n\n```bash\nharbor task init \"\u003Corg>\u002F\u003Ctask-name>\"\n```\n\nUseful flags:\n- `--description \"...\"`\n- `--author \"Jane Doe \u003Cjane@example.com>\"` (repeat for multiple authors)\n- `--no-pytest` — skip the pytest test template (use if planning Reward Kit or custom verifier)\n- `--no-solution` — skip solution\u002F directory\n- `--metadata-template path.toml` — pre-populate task.toml\n\nProduces:\n```\n\u003Ctask-name>\u002F\n├── instruction.md         # Task prompt for the agent\n├── task.toml              # Config and metadata\n├── environment\u002FDockerfile # Container definition\n├── solution\u002Fsolve.sh      # Reference solution (optional)\n└── tests\u002Ftest.sh          # Verifier script\n```\n\nIf the user wants a **multi-step task** (ordered steps with per-step\ninstructions, tests, and early stopping against a shared container), scaffold\nthe single-step layout first, then convert to the `steps\u002F` layout described in\nthe *Multi-step tasks* section below.\n\n## Step 2: Write instruction.md\n\nThis is the prompt the agent receives. Help the user write it clearly:\n\n- **State the goal concretely** — what file to create, what behavior to produce\n- **Specify expected outputs** — paths, formats, content\n- **Include constraints** — language, tools, approach\n- **Don't leak the tests** — describe what \"done\" looks like, not how you'll check it\n\nExample (from the ssh-key-pair tutorial):\n```markdown\n# SSH Key Pair Generation\n\nGenerate an SSH key pair in the files `~\u002F.ssh\u002Fid_rsa` and `~\u002F.ssh\u002Fid_rsa.pub`.\n\nDon't make them password protected.\n```\n\n## Step 3: Build the environment\n\nEdit `environment\u002FDockerfile` to install dependencies the task needs. The agent works \ninside this container.\n\n```dockerfile\nFROM ubuntu:24.04\nWORKDIR \u002Fapp\n\n# Install what the task requires — NOT the solution\nRUN apt-get update && apt-get install -y openssh-client && rm -rf \u002Fvar\u002Flib\u002Fapt\u002Flists\u002F*\n```\n\nFor multi-container setups, use `environment\u002Fdocker-compose.yaml` instead (note: most \ncloud sandbox providers only support Dockerfile).\n\n**Test the environment interactively** before writing the solution or tests:\n```bash\nharbor task start-env -p \"\u003Ctask-path>\" -e docker -a -i\n```\n\nThis is usually where task authors realize something is missing from the Dockerfile.\n\n## Step 4: Decide how to verify\n\n**This is the most important decision.** Ask the user: *\"How do you want to grade this \ntask?\"* Then help them pick:\n\nAlso ask: *\"Should the verifier run in the same environment as the agent, or in a\nseparate verifier environment?\"*\n\n- Use the default shared environment when tests need to inspect the agent's full\n  workspace, installed tools, or services.\n- Use a separate verifier environment when grading code, dependencies, API keys,\n  or OS requirements should stay hidden from the agent, or when verification\n  should run from a clean image.\n\nFor a separate verifier container, `tests\u002F` is the verifier image build context\nand the image must provide `\u002Ftests\u002Ftest.sh` (Linux) or `\u002Ftests\u002Ftest.bat`\n(Windows). Harbor copies `\u002Flogs\u002Fartifacts` and configured artifacts into the\nverifier environment, not the agent's whole workspace.\n\n```toml\n[verifier]\nenvironment_mode = \"separate\"\n\n[verifier.environment]\ndocker_image = \"ubuntu:24.04\"\n```\n\n### Option A: Reward Kit (recommended for most cases)\n\nUse when the verifier has multiple criteria, needs partial credit, uses an LLM\u002Fagent \njudge, or would benefit from composable reusable checks. See the `rewardkit` skill.\n\nGood fit signals:\n- Multiple things to check (file exists + content correct + command works)\n- Subjective quality dimensions (readability, correctness of prose)\n- Want partial credit rather than pass\u002Ffail\n- Want to compose built-ins like `file_contains`, `command_succeeds`, `json_key_equals`\n\n`tests\u002Ftest.sh`:\n```bash\n#!\u002Fbin\u002Fbash\nuvx --from 'harbor-rewardkit==0.1.*' rewardkit \u002Ftests\n```\n\nNote: the package is named `harbor-rewardkit` but the executable is `rewardkit`,\nhence `--from 'harbor-rewardkit==0.1.*' rewardkit`. Running\n`uvx harbor-rewardkit` directly will fail.\n\nThen add `tests\u002Fchecks.py` and\u002For `tests\u002Fjudge.toml`. Invoke the `rewardkit` skill to \ndesign the criteria.\n\n### Option B: pytest (good for deterministic unit-style checks)\n\nUse when the verification is straightforward assertion-style Python. Default template if \n`--no-pytest` wasn't passed.\n\n`tests\u002Ftest.sh`:\n```bash\n#!\u002Fbin\u002Fbash\napt-get update && apt-get install -y curl\ncurl -LsSf https:\u002F\u002Fastral.sh\u002Fuv\u002F0.9.7\u002Finstall.sh | sh\nsource $HOME\u002F.local\u002Fbin\u002Fenv\n\nuvx --with pytest==8.4.1 pytest \u002Ftests\u002Ftest_outputs.py\n\nif [ $? -eq 0 ]; then\n  echo 1 > \u002Flogs\u002Fverifier\u002Freward.txt\nelse\n  echo 0 > \u002Flogs\u002Fverifier\u002Freward.txt\nfi\n```\n\nExample `tests\u002Ftest_outputs.py`:\n```python\nfrom pathlib import Path\n\ndef test_file_exists():\n    assert (Path.home() \u002F \".ssh\" \u002F \"id_rsa\").exists()\n```\n\n### Option C: Custom shell\n\nFor simple single-command checks (e.g. a binary pass\u002Ffail from one command):\n```bash\n#!\u002Fbin\u002Fbash\nif diff -q \u002Fapp\u002Foutput.txt \u002Ftests\u002Fexpected.txt; then\n  echo 1 > \u002Flogs\u002Fverifier\u002Freward.txt\nelse\n  echo 0 > \u002Flogs\u002Fverifier\u002Freward.txt\nfi\n```\n\n### Reward file format (all options)\n\n- `\u002Flogs\u002Fverifier\u002Freward.txt` — single number (usually `0` or `1`)\n- `\u002Flogs\u002Fverifier\u002Freward.json` — `{\"accuracy\": 0.95, \"runtime_sec\": 1.2}` for multiple metrics\n\n**Always use absolute paths in `test.sh`.**\n\n## Step 5: Write the solution\n\nWrite `solution\u002Fsolve.sh` — a script that actually solves the task. The Oracle agent runs \nthis to sanity-check that the task is solvable and the tests pass on a correct solution.\n\n```bash\n#!\u002Fbin\u002Fbash\nssh-keygen -t rsa -f ~\u002F.ssh\u002Fid_rsa -N \"\"\n```\n\nMake it executable: `chmod +x solution\u002Fsolve.sh`.\n\n## Step 6: Configure task.toml\n\nWalk through the important fields:\n\n```toml\n[task]\nname = \"\u003Corg>\u002F\u003Ctask-name>\"\nversion = \"1.0.0\"\ndescription = \"One-line description\"\nkeywords = [\"jax\", \"mnist\", \"rewardkit\"]  # always populate — used for search\u002Ffiltering\n\n[metadata]\ndifficulty = \"easy\" | \"medium\" | \"hard\"\ncategory = \"programming\" | \"machine-learning\" | \"gpu\" | ...\ntags = [\"...\"]\n\n[agent]\ntimeout_sec = 120.0       # How long the agent has\n\n[verifier]\ntimeout_sec = 600.0       # How long tests have\n\n[environment]\nnetwork_mode = \"public\"   # Baseline at env start (defaults to public)\ncpus = 1                  # CPU cores\nmemory_mb = 2048          # RAM in MB\nstorage_mb = 10240        # Disk in MB\n```\n\n**Always populate `keywords`.** Pick 3–8 lowercase tokens covering the domain\n(language\u002Fframework\u002Fbenchmark family), the verifier style (`rewardkit`,\n`judge-grading`, `pytest`), and any notable hardware (`gpu`). They're surfaced in\n`harbor datasets list` and registry search.\n\n### Network policy\n\nNetwork access has three layers:\n\n1. **Baselines** — set when an environment starts, restored between phases\n2. **Phase overrides** — optional; only during `agent.run()` or `verify()`\n3. **Run-time merges** — `--allow-environment-host`, `--allow-agent-host` on `harbor run`\n\n| Field | Layer | When applied |\n| --- | --- | --- |\n| `[environment].network_mode` | Baseline | Agent env start; shared verifier uses this too |\n| `[verifier.environment].network_mode` | Baseline | Separate verifier env start |\n| `[agent].network_mode`, `[steps.agent].network_mode` | Override | During matching `agent.run()` |\n| `[verifier].network_mode`, `[steps.verifier].network_mode` | Override | During matching `verify()` |\n| `--allow-environment-host` | Run-time | Merged into `environment.extra_allowed_hosts` → `[environment]` baseline |\n| `--allow-agent-host` | Run-time | Merged into `agent.extra_allowed_hosts` → agent phase allowlist |\n\nModes: `public`, `no-network`, or `allowlist` with `allowed_hosts = [\"pypi.org\"]`\n(exact hostnames, IPv4\u002FIPv6 address literals or CIDR ranges, or leading wildcard hostnames, when supported by the selected environment; not URLs,\nports, or paths). Omitting `[environment].network_mode` defaults to `public`.\n\n`[agent]` \u002F `[verifier]` are **optional phase overrides** — only applied when set\n**and** different from the phase baseline. Matching the baseline is a no-op.\n\n**Shared verifier** (default): verifier runs in the agent container; baseline is\n`[environment]`. **Separate verifier**: baseline is `[verifier.environment]` if\nset, else a copy of `[environment]`.\n\n```toml\n# Agent starts offline; agent phase opens network; verifier stays offline\n[environment]\nnetwork_mode = \"no-network\"\n\n[agent]\nnetwork_mode = \"public\"\n\n[verifier]\nnetwork_mode = \"no-network\"\n```\n\nIf a phase override differs from its baseline, the environment provider must\nsupport `dynamic_network_policy` (E2B does; plain Docker does not). Prefer\n`environment_mode = \"separate\"` when agent and verifier need different baselines\nwithout runtime switching:\n\n```toml\n[environment]\nnetwork_mode = \"no-network\"\n\n[verifier]\nenvironment_mode = \"separate\"\n\n[verifier.environment]\nnetwork_mode = \"public\"   # Verifier baseline — not a phase override\n```\n\nRun-time host flags for eval jobs without editing `task.toml`:\n\n```bash\nharbor run -p \"\u003Ctask-path>\" -a oracle \\\n  --allow-environment-host pypi.org \\   # env start (Dockerfile apt\u002Fcurl)\n  --allow-agent-host files.pythonhosted.org   # agent.run() only\n```\n\nOn a `public` baseline, run-time host flags emit a warning and are ignored.\n\nExamples: `examples\u002Ftasks\u002Fnetwork-policy-matrix\u002F`. Full reference:\n`docs\u002Fcontent\u002Fdocs\u002Ftasks\u002Findex.mdx` (Network policy section).\n\nFor Reward Kit judges needing API keys:\n```toml\n[verifier.env]\nANTHROPIC_API_KEY = \"${ANTHROPIC_API_KEY}\"\n```\n\n## Step 7: Verify with the Oracle agent\n\n```bash\nharbor run -p \"\u003Ctask-path>\" -a oracle\n```\n\nOracle runs `solution\u002Fsolve.sh` and then the verifier. Reward should be `1.0`. If it's \nnot, debug in this order:\n1. Does `solve.sh` actually solve it? (`start-env -a -i` and run it manually)\n2. Does the verifier correctly detect success? (check `\u002Flogs\u002Fverifier\u002F` output)\n3. Are paths correct? (absolute vs relative)\n4. Are dependencies installed in the Dockerfile?\n\n## Step 8: Test with a real agent (optional)\n\n```bash\nharbor run -p \"\u003Ctask-path>\" -a terminus-2 -m anthropic\u002Fclaude-sonnet-4-6\n```\n\nIf the task is too easy (every model 1.0) or impossible (every model 0.0), consider \nadjusting difficulty.\n\n## Step 9: Update README.md (always the final step)\n\n`harbor task init` leaves `README.md` as a stub. Before wrapping up, populate it so\nfuture humans (and agents) can understand the task without reading every file.\nInclude:\n\n- **What the agent does** — one paragraph, link to `instruction.md`.\n- **Environment** — base image, key installed packages, cached data, hardware\n  (GPU\u002FCPU\u002FRAM), agent timeout.\n- **Verifier** — for Reward Kit tasks, a table of reward dimensions with type\n  (programmatic \u002F LLM judge \u002F agent judge) and what each measures; how they're\n  aggregated.\n- **Layout** — a tree of the task directory with one-line annotations.\n- **Running** — the concrete `harbor run` commands (Oracle + real agent), with the\n  right provider flag if the task needs a GPU.\n\nTreat this as docs, not marketing — the reader wants to know *what they'd need to\nchange* to modify the task.\n\n## Multi-step tasks\n\nUse when the work splits into ordered phases that should be scored separately,\nwhen you want early stopping between phases, or when you're testing an agent's\nability to build on its own prior work. Steps share one container; files\npersist across steps.\n\n### Directory layout\n\nReplace the task-root `instruction.md`, `tests\u002F`, and `solution\u002F` with a\n`steps\u002F` directory containing one sub-directory per step:\n\n```\n\u003Ctask-name>\u002F\n├── task.toml\n├── environment\u002FDockerfile       # Built once, shared across all steps\n├── steps\u002F\n│   ├── scaffold\u002F\n│   │   ├── instruction.md       # Prompt for this step\n│   │   ├── workdir\u002F             # Uploaded to WORKDIR before the agent runs\n│   │   │   └── setup.sh         # Optional pre-agent hook (reserved filename)\n│   │   ├── tests\u002Ftest.sh        # Per-step verifier\n│   │   └── solution\u002Fsolve.sh    # Per-step Oracle solution (optional)\n│   ├── implement\u002F\n│   │   └── ...\n│   └── document\u002F\n│       └── ...\n└── tests\u002F                       # Optional shared helpers + fallback test.sh\n```\n\nTask-level `tests\u002F` is uploaded to `\u002Ftests` for each step's verification, then\nthe step's own `tests\u002F` is layered on top (same-name files win). Use this for\nshared helpers.\n\n`steps\u002F{name}\u002Fworkdir\u002Fsetup.sh` is a **reserved filename**: if present, it runs\nafter the `workdir\u002F` upload and before the agent, as the step's agent user,\nwith cwd = WORKDIR. Non-zero exit aborts the step and the trial. Have it\n`rm -- \"$0\"` on its last line if the agent shouldn't see it.\n\n### task.toml\n\n```toml\nschema_version = \"1.4\"\n\n[task]\nname = \"\u003Corg>\u002F\u003Ctask-name>\"\nversion = \"1.0.0\"\n\n# How per-step rewards roll up into the trial-level verifier_result.\n# \"mean\" (default): per-key mean across steps that produced a result.\n# \"final\": the last step's verifier_result verbatim.\nmulti_step_reward_strategy = \"mean\"\n\n[[steps]]\nname = \"scaffold\"              # Must match the directory under steps\u002F\nmin_reward = 1.0               # Abort trial if this step's reward \u003C 1.0\n[steps.agent]\ntimeout_sec = 60.0             # Overrides task-level [agent].timeout_sec\n[steps.verifier]\ntimeout_sec = 30.0\n\n[[steps]]\nname = \"implement\"\n# Dict form gates on specific keys from a multi-dim reward:\nmin_reward = { correctness = 0.8, style = 0.5 }\n[steps.agent]\ntimeout_sec = 120.0\n[steps.verifier]\ntimeout_sec = 30.0\n\n[[steps]]\nname = \"document\"\n[steps.agent]\ntimeout_sec = 60.0\n[steps.verifier]\ntimeout_sec = 30.0\n```\n\nPer-step overrides available: `agent.timeout_sec`, `agent.user`,\n`agent.network_mode`, `verifier.timeout_sec`, `verifier.env`, `verifier.user`,\n`verifier.network_mode`, `verifier.environment_mode`, `verifier.environment`,\n`steps.verifier.environment.network_mode`, `healthcheck.*`, `artifacts`. Unset\nfields fall back to the task-level values.\n\n### Choosing a reward strategy\n\n- **`\"mean\"`** — aggregate signal across all steps; good for continuous\n  progress rewards.\n- **`\"final\"`** — last step's verifier_result is the trial reward. Right when\n  the final step is an end-to-end check whose dict already represents the full\n  task. Caveat: if `min_reward` triggers an early abort, `\"final\"` uses the\n  *aborted* step's result, not the intended final step.\n\n### Artifacts\n\nStep-level `artifacts` are collected into `steps\u002F{name}\u002Fartifacts\u002F` after that\nstep's verification. Task-level and trial-level artifacts are collected at\nevery step in addition to the step-level ones.\n\n### Oracle verification\n\n`harbor run -p \"\u003Ctask-path>\" -a oracle` runs each step's `solution\u002Fsolve.sh`,\nthen each step's verifier, in order. Trial reward should be `1.0` across the\naggregation strategy.\n\n### Full reference + worked example\n\n- Docs: `docs\u002Fcontent\u002Fdocs\u002Ftasks\u002Fmulti-step.mdx`\n- Example task: `examples\u002Ftasks\u002Fhello-multi-step-advanced\u002F`\n\n## Special features (mention if relevant)\n\n- **Network policy**: Baselines on `[environment]` \u002F `[verifier.environment]`; phase\n  overrides on `[agent]` \u002F `[verifier]`; see *Network policy* under Step 6\n- **MCP servers**: Add `[[environment.mcp_servers]]` in task.toml for agent tooling\n- **Healthcheck**: Add `[environment.healthcheck]` for services that need to be ready\n- **GPU**: Set `environment.gpus` and optionally `environment.gpu_types`\n- **Pre-built image**: Set `environment.docker_image` instead of building from Dockerfile. You can omit `environment\u002FDockerfile` and place runtime files (configs, scripts, data) directly under `environment\u002F`; Harbor uploads them into the container workdir when the environment starts.\n- **Non-root user**: Set `agent.user` \u002F `verifier.user` for isolation\n\n## Common pitfalls\n\n- Forgetting to write the reward file → task \"passes\" silently with reward 0\n- Using relative paths in `test.sh` → breaks when Harbor runs it from a different cwd\n- Installing the solution into the Dockerfile → agent already gets the answer\n- Test script leaks into `instruction.md` → agent sees the rubric and gaming becomes trivial\n- Forgetting `chmod +x solution\u002Fsolve.sh` → Oracle agent fails\n- Leaving `keywords = []` in task.toml → task is invisible to registry search\n- Leaving `README.md` as a stub → teammates have no way to understand the task at a glance\n- Putting `network_mode` on `[agent]` expecting it to apply at env start → use\n  `[environment].network_mode` for the baseline; agent\u002Fverifier fields are phase overrides\n- Phase override differs from baseline on Docker → task rejected unless provider supports\n  `dynamic_network_policy`; use separate verifier env or match the baseline instead\n",{"name":5,"description":7,"argument-hint":44},[45],"org\u002Ftask-name","main","10c78a87e1f92695d78d15b1464a6107dcac8837","deps\u002Fharbor\u002Fskills\u002Fcreate-task",9,"https:\u002F\u002Fgithub.com\u002Fpku-liang\u002Fhwe-bench","https:\u002F\u002Fgithub.com\u002Fpku-liang","https:\u002F\u002Fericlyun.github.io\u002F","2026-11-11 10:00:32",{"source":8,"skillName":5,"commands":55,"prompt":65,"zipAvailable":66},[56,59,62],{"tool":57,"command":58},"npx","npx skills add pku-liang\u002Fhwe-bench --skill create-task",{"tool":60,"command":61},"bunx","bunx skills add pku-liang\u002Fhwe-bench --skill create-task",{"tool":63,"command":64},"pnpm","pnpm dlx skills add pku-liang\u002Fhwe-bench --skill create-task","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpku-liang\u002Fhwe-bench\u002Fblob\u002Fmain\u002Fdeps\u002Fharbor\u002Fskills\u002Fcreate-task\u002FSKILL.md",true,[],[69,92,108,122,143,165],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":39},"01M71FNJ6S5MATHHD81PMD42MM","upload-parity-experiments","Create or reuse Hugging Face dataset PRs for `harborframework\u002Fparity-experiments` and upload Harbor parity\u002Foracle result folders efficiently with sparse checkout, raw git pushes, and Git LFS.","https:\u002F\u002Fgithub.com\u002Fpku-liang\u002Fhwe-bench\u002Fblob\u002Fmain\u002Fdeps\u002Fharbor\u002Fskills\u002Fupload-parity-experiments\u002FSKILL.md",[],[76,77,78,79,80,81,82,83,84,85,86,87,88,26,89,90,91,21],"upload","upload parity experiments","experiments","parity","face","checkout","with","harbor","dataset","oracle","reuse","prs","lfs","result","git","hugging face",{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":39},"01M71FNG0TQD738NZGZN1W31AF","rewardkit","Write Harbor task verifiers using Reward Kit. Use when creating or editing a task's tests\u002F directory, adding grading criteria, setting up LLM\u002Fagent judges, or designing verifiers that produce a reward score.","https:\u002F\u002Fgithub.com\u002Fpku-liang\u002Fhwe-bench\u002Fblob\u002Fmain\u002Fdeps\u002Fharbor\u002Fskills\u002Frewardkit\u002FSKILL.md",[],[94,19,99,31,22,100,34,101,102,103,104,83,105,106,107],"directory","produce","tests","up","editing","setting","grading","write","Verifiers",{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":39},"01M71FNCKK74MFE7MYSPVN0YAT","publish","Publish a Harbor task or dataset to the registry. Use when the user wants to upload, publish, or share tasks or datasets\u002Fbenchmarks on the Harbor registry.","https:\u002F\u002Fgithub.com\u002Fpku-liang\u002Fhwe-bench\u002Fblob\u002Fmain\u002Fdeps\u002Fharbor\u002Fskills\u002Fpublish\u002FSKILL.md",[],[110,115,83,116,19,117,118,119,84,31,22,120,121,76],"share","path","user","benchmarks","tasks","registry","datasets",{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":126,"categories":127,"keywords":128,"publishedAt":39},"01M71FN9MJMRMA5E1AZCRFKN6M","harbor-exec","Use when working with Harbor's `harbor exec` CLI workflow: compiling files, directories, or globs into Harbor tasks; running map jobs; configuring artifacts and existence-only verification; using map-reduce; writing or reviewing ExecConfig YAML\u002FJSON\u002FTOML; or debugging command behavior, config validation, and job outputs.","https:\u002F\u002Fgithub.com\u002Fpku-liang\u002Fhwe-bench\u002Fblob\u002Fmain\u002Fdeps\u002Fharbor\u002Fskills\u002Fharbor-exec\u002FSKILL.md",[],[129,83,130,131,132,133,134,135,136,82,119,137,138,31,139,22,32,140,141,142],"exec","harbor exec","toml","directories","running","outputs","artifacts","verification","workflow","reduce","debugging","files","map","only",{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":154,"publishedAt":39},"01M71FN46QH5B25X9SXCJ5PNWD","create-adapter","Scaffold a new Harbor benchmark adapter by running `harbor adapter init` and then guide implementation using the Adapters Agent Guide as the authoritative spec.","https:\u002F\u002Fgithub.com\u002Fpku-liang\u002Fhwe-bench\u002Fblob\u002Fmain\u002Fdeps\u002Fharbor\u002Fskills\u002Fcreate-adapter\u002FSKILL.md",[149],{"categoryNo":150,"categoryName":151,"taxonomyType":152,"taxonomyTypeDesc":153},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[21,155,156,157,158,159,160,161,162,29,83,33,163,164,133],"create adapter","adapter","spec","scaffold","guide","agent guide","Adapters","implementation","init","as",{"skillNo":166,"name":167,"nameZh":6,"description":168,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":169,"categories":170,"keywords":171,"publishedAt":39},"01M71FN35PFRFKD7ZKMH9WHZDV","generate-greeting","Generate a greeting message and write it to a file.","https:\u002F\u002Fgithub.com\u002Fpku-liang\u002Fhwe-bench\u002Fblob\u002Fmain\u002Fdeps\u002Fharbor\u002Fexamples\u002Ftasks\u002Fhello-skills\u002Fenvironment\u002Fskills\u002Fgenerate-greeting\u002FSKILL.md",[],[172,173,174,106,175,176,177],"greeting","generate greeting","generate","it","file","message"]