bend-harness-qa
Drive a live bend-harness session programmatically with bend_client.py to test features end to end, reproduce and investigate bugs, and write precise bug reports. Use when asked to test, verify, QA, benchmark, or investigate the Bend harness.
- License 许可
- Apache-2.0
- 所属仓库
- gvergnaud/bise
- 最近更新
- 2026-11-15
- Stars
- ★ 51
- Forks
- 9
01 前言 Front Matter
- name
- bend-harness-qa
- description
- Drive a live bend-harness session programmatically with bend_client.py to test features end to end, reproduce and investigate bugs, and write precise bug reports. Use when asked to test, verify, QA, benchmark, or investigate the Bend harness.
02 资源文件 2
- 参考文档reference/client-api.md3.8 KB
- 脚本scripts/e2e_template.py2.1 KB
03 SKILL.md 正文
Bend Harness QA
You control a REAL live harness session (GLM model + bash + MCP tools + skills) from Python: the same wire protocol the TUI speaks, but scripted and reproducible. This is how you test a new feature end to end, reproduce a user-reported bug, or investigate weird behavior.
Repo: /Users/gabrielvergnaud/lab/bend-lab/harness (all relative paths below are inside it).
Golden rules
- ALWAYS run with
python3 -u(unbuffered stdout) — without it a long turn looks dead. - One TCP connection per session: the REPL serves one client at a time. Sequential
BendSession.fresh()calls are fine (each picks a private port); alwayss.close()when done (the REPL dies with it, the session checkpoint stays). - A turn ends at the
--- idleline.say()blocks until then (default timeout 900s — pass a shorter one for smoke tests). - Bash tool calls slower than
BEND_BG_AFTERseconds (default 30) hand off to a background slot instead of blocking. Usefresh(bg_after=N)to shrink the window in tests.
Minimal run
import sys; sys.path.insert(0, "scripts") # run from the repo root
from bend_client import BendSession
s = BendSession.fresh() # or BendSession.resume(path)
try:
lines = s.say("reply with exactly OK", timeout=120)
assert "OK" in s.last_assistant(lines)
print("PASS")
finally:
s.close()
Run it: python3 -u /tmp/mytest.py. The full turn transcript is in lines (print it when debugging); the client log is /tmp/bend-client-<port>.log.
For anything beyond the basics, read reference/client-api.md (full client API + the wire events) and copy scripts/e2e_template.py as a starting point.
E2E recipes
- Feature smoke test: one turn, instruct precisely, then assert on the obs lines (tool started/finished, tool_result) AND
last_assistant(). Print PASS/FAIL, never eyeball. - Mid-turn steering: run
say()in a thread, wait for the tool call to appear in the log, thens.steer_midturn(text)— it writes the file side-channel/tmp/bend-steer-<port>.txt. Assert: an obssteering_received:mid-turn, an obssteered:(injected into the next model input), and the final answer honoring the steered text. (s.steer()sends on the SOCKET — it is only read at idle and starts a NEW turn; that is NOT mid-turn steering.) - Background commands:
BendSession.fresh(bg_after=2), ask the agent to runsleep 5 && echo done. Assert the tool result carries the handoff contract (slot id, tail/stdin/status/kill commands) and that a second bash call can read the slot's.out/.rcunder/tmp/bend-bg-<port>/. - Skills: ask the model to call the
skilltool with exact args{"name":"<exact>"}. Assert the result starts with the SKILL.md CONTENT (frontmatter stripped), and that a bogus name returns "Skill ... is not available" plus the available list. - Sessions / CLI flags (not client-driven):
./run.sh --headless [--scripted](what bend_client.py runs: one READY line, lives until stdin closes; there is no single-agent TUI any more, BISE-113). Sessions live in~/.bend-harness/sessions/<id>.txt;--continue= latest mtime,--resume <id-or-unique-prefix>, errors list the available ids.
Before you report a bug
Separate a harness bug from an environment/model flake — run the deterministic gates from the repo root:
python3 tests/proof_run.py . 4 "$TMPDIR/proofs" # laws: every bend/proofs/*.bend (NEVER `bend update`: pin 2.0.27)
bend bend/runtime/demo.bend -o harness-demo && ./harness-demo > /tmp/demo-a.txt && ./harness-demo > /tmp/demo-b.txt && diff /tmp/demo-a.txt /tmp/demo-b.txt # deterministic
~/.bend/bin/bend bend/tests/test-bg.bend -o /tmp/bg-suite && rm -rf /tmp/bgtest && BEND_BG_ROOT=/tmp/bgtest BEND_BG_AFTER=2 /tmp/bg-suite
A live-model failure with all gates green is a live-integration bug or a model flake — re-run the e2e once before reporting.
Bug report format
Write reports as markdown (scratchpad or the file the user asked for), one report per bug:
- Title — one line, the symptom.
- Environment — commit hash (
git log --oneline -1), mode (live/scripted), env vars (BEND_BG_AFTER, port). - Repro — the SMALLEST python script that triggers it (must be runnable with
python3 -u), or the exact pipe command. - Expected / Observed — two short lines, not prose.
- Transcript excerpt — the relevant obs lines only, not the whole turn.
- Hypothesis — the suspected module (
runtime/main.bend,runtime/bash.bend,runtime/provider.bend,core/session.bend, the TUI...) and what the gates say. - Severity — blocks-turn / degrades / cosmetic.
Investigation pointers
- Session checkpoints:
/tmp/bend-sessions/session-<port>.txt(client-spawned) or~/.bend-harness/sessions/(binary-spawned). Format: the checkpoint text protocol (core/checkpoint.bend). - The harness can test ITSELF: via
say(), ask the live agent to exercise its own tools (bash, background slots, skills, programmatic calling) and return a structured report — it knows its own contract. That is the QA-round pattern; parse its final answer for findings. - Tool routing:
bend/runtime/main.bend(exec_picktop-level,exec_program.callfrom the sandbox). Pure contracts are pinned inbend/laws/*.bend, each closed by itsbend/proofs/*.bend. - MCP index
~/.bend-harness/mcp-index.txt, skills index~/.bend-harness/skills-index.txt.
Gotchas
- No
timeoutcommand on macOS (GNU coreutils absent) — use the client timeout args. - Assistant text escapes newlines as literal
\non the wire — read it vialast_assistant()/unescape(). - The client log shows
REPL onwhen the REPL is ready; a TCP probe would steal the resume greeting — never probe, wait for the banner. - A client-spawned session is LIVE (real model, real bash): commands the agent runs are real commands on this machine.
04 同仓库更多 Skills
greet
Greet someone the hello-plugin way. Use when asked for a hello-plugin greeting.
triage
triage Linear issues
jot
jot a note
bise-demo
Run bise's built-in demo, a short role-play where a small product team (a pm, a designer, a dev) starts, talks like colleagues, asks the user one question and one agent is dropped, with no file or git change. Use when the user asks to see what bise or you can do, for a demo, a tour or your features (show me what you can do, give me a demo, what can you do?, demo all your features), in any language.
bise-pages
暂无简介
slides
Make, read or edit a PowerPoint deck (.pptx) - a review, a pitch, a status deck, slides with tables and charts - with a TypeScript script run by `bise office run`. Use when the user asks for slides, a deck or a pptx, sends one, or leaves notes on a pptx artifact.