stress-test
Full stress-test suite for stablehand — exercises tools, chaining, HITL, multi-model, code execution
- License 许可
- MIT
- 最近更新
- 2026-10-25
- Stars
- ★ 15
- Forks
- 2
01 前言 Front Matter
- name
- stress-test
- description
- Full stress-test suite for stablehand — exercises tools, chaining, HITL, multi-model, code execution
03 SKILL.md 正文
Full stress test plan
API reference (quick)
| Endpoint | Method | Returns |
|---|---|---|
/api/health |
GET | { status: "ok", ... } |
/api/agents |
GET | [ { id, name, harness, ... }, ... ] |
/api/agents/:id/message |
POST { content } |
{ sessionId, messageId, conversationId } |
/api/agents/:id/session |
POST { task } |
{ sessionId } (background run) |
/api/agents/:id/sessions |
GET | { sessions: [{ id, status, ... }] } |
/api/agents/:id/conversations |
GET | [ { id, agentId, ... } ] (plain array) |
/api/agents/:id/conversations/:id/messages |
GET | [ { id, role, content, ... } ] |
/api/agents/:id/interrupts |
GET | { interrupts: [{ sessionId, toolName, ... }] } |
/api/agents/:id/interrupt |
POST { action, sessionId } |
{ resolved: true, ... } |
/api/settings |
GET | { maxGlobalContainers, ... } (13 keys, no wrapper) |
/api/settings |
PUT { key: value } |
Updated settings object |
/api/events |
GET ?type=...&limit=... |
{ events: [...] } |
/api/triggers/emit |
POST { type, payload } |
{ ok: true } |
UI routes (SPA):
/— Dashboard (agents list)/agents/:id— Agent detail (tabs: Overview, Sessions, Tools, etc.)/agents/:id/conversations/new— New chat conversation/agents/:id/conversations/:conversationId— Existing conversation/events,/tools,/flow,/knowledge,/triggers,/artifacts,/settings
Prerequisites
The full stack must be running with the sandbox profile before starting.
docker compose up --build -d
Verify readiness:
# API responds
curl -sf http://localhost:3002/api/agents > /dev/null && echo "API: OK" || echo "API: DOWN"
# Egress proxy is healthy (required for sandbox → LLM API connectivity)
docker compose ps egress-proxy --format '{{.Status}}' | grep -q healthy && echo "Proxy: OK" || echo "Proxy: DOWN"
# MinIO is running (required for artifact tests)
curl -sf http://localhost:9000/minio/health/live > /dev/null 2>&1 && echo "MinIO: OK" || echo "MinIO: DOWN — artifact tests will be skipped"
# Sandbox harness images exist
for img in langgraph claude-agents pi-agent; do
docker image inspect "stablehand/harness-${img}:latest" > /dev/null 2>&1 && echo "Image ${img}: OK" || echo "Image ${img}: MISSING — run: docker compose up --build"
done
All checks must pass before proceeding. Without the sandbox profile, agents cannot reach external LLM APIs and tool proxying (emit-event, request-approval) will fail silently.
Known issues
HITL lifecycle conflict (BUG-1): RESOLVED (2026-04-07). Sessions now complete after interrupt approval. The
stress.hitl.completeevent is reliably emitted. The original issue — lifecycle policy destroying the container duringrequest-approvalblocking — has been fixed.
Event chain agent compliance (BUG-2): The trigger mechanism works correctly (trigger fires, chain-responder session spawns and completes). However, the chain-responder agent does not reliably call
emit-eventwithstress.chain.done. This is an LLM instruction-following issue, not a system bug. The chain plumbing itself is verified working.
Shared setup
set -euo pipefail
API_BASE="http://localhost:3002"
agent_id() {
local name="$1"
curl -s "$API_BASE/api/agents" | jq -r --arg name "$name" 'first(.[] | select(.name == $name) | .id)'
}
# POST /api/agents/:id/message returns { sessionId, messageId, conversationId }
send_message() {
local agent_id="$1"
local content="$2"
curl -s -X POST "$API_BASE/api/agents/$agent_id/message" \
-H 'Content-Type: application/json' \
-d "$(jq -nc --arg content "$content" '{content: $content}')" \
| jq -r '.sessionId'
}
# GET /api/agents/:id/sessions returns { sessions: [...] }
wait_for_session() {
local agent_id="$1"
local session_id="$2"
local ok_statuses="$3"
local timeout="${4:-180}"
local elapsed=0
while true; do
status="$(curl -s "$API_BASE/api/agents/$agent_id/sessions" \
| jq -r --arg sid "$session_id" 'first(.sessions[]? | select(.id == $sid) | .status) // empty')"
echo "session=$session_id status=${status:-missing} (${elapsed}s)"
[[ " $ok_statuses " == *" $status "* ]] && return 0
[[ "$status" =~ ^(failed|cancelled|rejected)$ ]] && return 1
# HITL: session may error due to container lifecycle but interrupt is persisted
if [[ "$status" == "error" ]]; then
has_interrupt="$(curl -s "$API_BASE/api/agents/$agent_id/interrupts" \
| jq --arg sid "$session_id" '[.interrupts[]? | select(.sessionId == $sid)] | length')"
if [[ "$has_interrupt" -gt 0 ]]; then
echo " session errored but has pending interrupt (durable HITL)"
return 0
fi
return 1
fi
((elapsed+=3))
[[ $elapsed -ge $timeout ]] && { echo "TIMEOUT after ${timeout}s"; return 1; }
sleep 3
done
}
# Wait for a pending interrupt to appear for a session
wait_for_interrupt() {
local agent_id="$1"
local session_id="$2"
local timeout="${3:-120}"
local elapsed=0
while true; do
interrupt="$(curl -s "$API_BASE/api/agents/$agent_id/interrupts" \
| jq --arg sid "$session_id" 'first(.interrupts[]? | select(.sessionId == $sid))')"
[[ "$interrupt" != "null" ]] && [[ -n "$interrupt" ]] && { echo "$interrupt"; return 0; }
((elapsed+=3))
echo " waiting for interrupt... (${elapsed}s)" >&2
[[ $elapsed -ge $timeout ]] && { echo "TIMEOUT waiting for interrupt" >&2; return 1; }
sleep 3
done
}
# POST /api/agents/:id/interrupt accepts { action, sessionId }
resolve_interrupt() {
local agent_id="$1"
local session_id="$2"
local action="$3"
curl -s -X POST "$API_BASE/api/agents/$agent_id/interrupt" \
-H 'Content-Type: application/json' \
-d "$(jq -nc --arg action "$action" --arg sessionId "$session_id" '{action: $action, sessionId: $sessionId}')"
}
Phase 1: Seed all agents
npx tsx tests/stress/seed.ts
LG_OPENAI_ID="$(agent_id stress-lg-openai)"
LG_GEMINI_ID="$(agent_id stress-lg-gemini)"
PI_OPENAI_ID="$(agent_id stress-pi-openai)"
PI_GEMINI_ID="$(agent_id stress-pi-gemini)"
CLAUDE_ID="$(agent_id stress-claude-sonnet)"
CHAIN_ID="$(agent_id stress-chain-responder)"
HITL_ID="$(agent_id stress-hitl)"
Expected:
- 7 stress agents imported
- trigger registered:
stress.tools.complete -> stress-chain-responder
Pass/fail:
- Pass if every id above is non-empty
- Fail if seed exits non-zero or any id is missing
Phase 2: Tool execution (stress-lg-openai)
Run the LangGraph agent with GPT-5.4 and verify it can call emit-event.
LG_OPENAI_SID="$(send_message "$LG_OPENAI_ID" "Run the stress test now. Confirm you are operational and emit stress.tools.complete.")"
wait_for_session "$LG_OPENAI_ID" "$LG_OPENAI_SID" "completed error"
curl -s "$API_BASE/api/events?type=stress.tools.complete&agentId=$LG_OPENAI_ID&limit=20" \
| jq '.events | length'
Expected:
- Session reaches
completedafter emitting the event stress.tools.completeevent exists for this agent
Pass/fail:
- Pass if event count > 0
- Fail if no completion event is persisted
Phase 3: Multi-model validation (stress-lg-gemini, stress-pi-gemini)
LangGraph + Gemini
LG_GEMINI_SID="$(send_message "$LG_GEMINI_ID" "Verify Gemini LangGraph execution and emit stress.gemini.complete.")"
wait_for_session "$LG_GEMINI_ID" "$LG_GEMINI_SID" "completed"
curl -s "$API_BASE/api/events?type=stress.gemini.complete&limit=20" | jq '.events | length'
PI Agent + Gemini
PI_GEMINI_SID="$(send_message "$PI_GEMINI_ID" "Verify Gemini PI-agent execution and emit stress.pi-gemini.complete.")"
wait_for_session "$PI_GEMINI_ID" "$PI_GEMINI_SID" "completed"
curl -s "$API_BASE/api/events?type=stress.pi-gemini.complete&limit=20" | jq '.events | length'
Expected:
- Both Gemini-backed agents complete successfully
- Each emits its model-specific event
Pass/fail:
- Pass if both sessions are
completedand both event counts are > 0 - Fail if either session errors or either event is missing
Phase 4: Event chain (stress-lg-openai -> stress-chain-responder)
Emit the trigger event externally, then verify the responder runs.
CHAIN_BEFORE="$(curl -s "$API_BASE/api/agents/$CHAIN_ID/sessions" | jq '.sessions | length')"
curl -s -X POST "$API_BASE/api/triggers/emit" \
-H 'Content-Type: application/json' \
-d '{"type":"stress.tools.complete","payload":{"source":"stress-test-skill","ts":0}}' \
| jq .
# Wait for a new session to appear
timeout=60 elapsed=0
while true; do
CHAIN_SID="$(curl -s "$API_BASE/api/agents/$CHAIN_ID/sessions" \
| jq -r --argjson before "$CHAIN_BEFORE" 'if (.sessions | length) > $before then .sessions[0].id else empty end')"
[[ -n "$CHAIN_SID" ]] && break
((elapsed+=2))
[[ $elapsed -ge $timeout ]] && { echo "TIMEOUT"; break; }
sleep 2
done
wait_for_session "$CHAIN_ID" "$CHAIN_SID" "completed"
curl -s "$API_BASE/api/events?type=stress.chain.done&limit=20" | jq '.events | length'
Expected:
- Trigger emission returns 200
stress-chain-responderreceives a new sessionstress.chain.doneis persisted
Pass/fail:
- Pass if a new responder session appears and completes
- Fail if the trigger emits but no responder session starts
Phase 5: Code execution (stress-claude-sonnet)
CLAUDE_SID="$(send_message "$CLAUDE_ID" "Write the stress test file now, verify it exists, and emit stress.claude.complete.")"
wait_for_session "$CLAUDE_ID" "$CLAUDE_SID" "completed"
curl -s "$API_BASE/api/events?type=stress.claude.complete&limit=20" | jq '.events | length'
Expected:
- Claude agent completes
- event proves the write/verify workflow finished
Pass/fail:
- Pass if session is
completedand event count > 0 - Fail otherwise
Phase 6: HITL (stress-hitl) — approve, then reject
HITL lifecycle is fully functional. Sessions survive the interrupt and complete after approval/rejection.
Approve path
HITL_APPROVE_SID="$(send_message "$HITL_ID" "Request approval for writing stress test results, then emit stress.hitl.complete.")"
# Wait for interrupt to appear (session may show running or error)
INTERRUPT="$(wait_for_interrupt "$HITL_ID" "$HITL_APPROVE_SID")"
echo "$INTERRUPT" | jq '{id, toolName, status}'
# Approve
resolve_interrupt "$HITL_ID" "$HITL_APPROVE_SID" "approve" | jq .
# Check resolution
curl -s "$API_BASE/api/agents/$HITL_ID/interrupts" \
| jq --arg sid "$HITL_APPROVE_SID" '[.interrupts[]? | select(.sessionId == $sid)] | length'
Reject path
HITL_REJECT_SID="$(send_message "$HITL_ID" "Request approval again — this one will be rejected.")"
INTERRUPT="$(wait_for_interrupt "$HITL_ID" "$HITL_REJECT_SID")"
echo "$INTERRUPT" | jq '{id, toolName, status}'
resolve_interrupt "$HITL_ID" "$HITL_REJECT_SID" "reject" | jq .
curl -s "$API_BASE/api/agents/$HITL_ID/interrupts" \
| jq --arg sid "$HITL_REJECT_SID" '[.interrupts[]? | select(.sessionId == $sid)] | length'
Expected:
- Both runs produce a durable interrupt with
request-approvaltool - Approve and reject both resolve the interrupt (pending count → 0)
- Sessions complete normally after approval;
stress.hitl.completeevent emitted on approve path
Pass/fail:
- Pass if both interrupts are persisted and both resolutions clear the pending list
- Fail if interrupts are not persisted or resolution fails
Phase 7: Verify events persisted in /api/events
curl -s "$API_BASE/api/events?limit=200" | jq '[.events[].type] | unique'
Confirm these event types exist:
stress.tools.completestress.gemini.completestress.pi-gemini.completestress.claude.completestress.hitl.completestress.chain.done(unreliable — see BUG-2; chain mechanism works but agent may not emit)
Pass/fail:
- Pass if the first 5 types are present in persisted events
stress.chain.doneis best-effort due to BUG-2 (LLM compliance)- Fail if any of the first 5 event types is missing
Phase 8: System settings API
curl -s "$API_BASE/api/settings" | jq .
SETTINGS_COUNT="$(curl -s "$API_BASE/api/settings" | jq 'keys | length')"
ORIGINAL_MAX_CONCURRENT="$(curl -s "$API_BASE/api/settings" | jq -r '.maxConcurrentSessionsPerAgent')"
curl -s -X PUT "$API_BASE/api/settings" \
-H 'Content-Type: application/json' \
-d '{"maxConcurrentSessionsPerAgent":4}' \
| jq .
INVALID_SETTINGS_RESPONSE="$(curl -s -w $'\n%{http_code}' -X PUT "$API_BASE/api/settings" \
-H 'Content-Type: application/json' \
-d '{"maxConcurrentSessionsPerAgent":-1}')"
INVALID_SETTINGS_STATUS="${INVALID_SETTINGS_RESPONSE##*$'\n'}"
INVALID_SETTINGS_BODY="${INVALID_SETTINGS_RESPONSE%$'\n'*}"
echo "$INVALID_SETTINGS_BODY" | jq .
CURRENT_MAX_CONCURRENT="$(curl -s "$API_BASE/api/settings" | jq -r '.maxConcurrentSessionsPerAgent')"
curl -s -X PUT "$API_BASE/api/settings" \
-H 'Content-Type: application/json' \
-d "$(jq -nc --argjson value "$ORIGINAL_MAX_CONCURRENT" '{maxConcurrentSessionsPerAgent: $value}')" \
| jq .
Expected:
GET /api/settingsreturns all 13 settingsPUT /api/settingspersists a valid update tomaxConcurrentSessionsPerAgent- invalid update returns 400 and does not overwrite the valid value
- final restore returns the original value
Pass/fail:
- Pass if
SETTINGS_COUNT=13,INVALID_SETTINGS_STATUS=400, andCURRENT_MAX_CONCURRENT=4before restore - Fail if any settings key is missing, validation accepts the negative value, or the round-trip update does not persist
Phase 9: Artifact storage
# MinIO must be running for artifact tests
if curl -sf http://localhost:9000/minio/health/live > /dev/null 2>&1; then
echo "MinIO: OK"
ARTIFACT_PAYLOAD="stress-test-artifact-payload"
ARTIFACT_UPLOAD_RESPONSE="$(curl -s -X POST "$API_BASE/api/artifacts/upload-url" \
-H 'Content-Type: application/json' \
-d "$(jq -nc --arg agentId "$LG_OPENAI_ID" --arg name "stress-test-artifact.txt" --arg mimeType "text/plain" --argjson sizeBytes 100 '{agentId: $agentId, name: $name, mimeType: $mimeType, sizeBytes: $sizeBytes}')")"
echo "$ARTIFACT_UPLOAD_RESPONSE" | jq .
ARTIFACT_ID="$(echo "$ARTIFACT_UPLOAD_RESPONSE" | jq -r '.artifactId')"
ARTIFACT_UPLOAD_URL="$(echo "$ARTIFACT_UPLOAD_RESPONSE" | jq -r '.uploadUrl')"
curl -s -X PUT "$ARTIFACT_UPLOAD_URL" \
-H 'Content-Type: text/plain' \
--data-binary "$ARTIFACT_PAYLOAD" \
> /dev/null
# List: GET /api/artifacts?agentId=... returns { artifacts: [...] }
curl -s "$API_BASE/api/artifacts?agentId=$LG_OPENAI_ID" | jq '.artifacts | length'
# Get single: GET /api/artifacts/:id returns the raw Artifact object (no wrapper)
curl -s "$API_BASE/api/artifacts/$ARTIFACT_ID" | jq '{id, name, mimeType, sizeBytes}'
ARTIFACT_DOWNLOAD_RESPONSE="$(curl -s "$API_BASE/api/artifacts/$ARTIFACT_ID/download-url")"
ARTIFACT_DOWNLOAD_URL="$(echo "$ARTIFACT_DOWNLOAD_RESPONSE" | jq -r '.url')"
DOWNLOADED_ARTIFACT_PAYLOAD="$(curl -s "$ARTIFACT_DOWNLOAD_URL")"
curl -s -X DELETE "$API_BASE/api/artifacts/$ARTIFACT_ID" | jq .
REMAINING_ARTIFACTS="$(curl -s "$API_BASE/api/artifacts?agentId=$LG_OPENAI_ID" | jq --arg artifact_id "$ARTIFACT_ID" '[.artifacts[]? | select(.id == $artifact_id)] | length')"
else
echo "MinIO: DOWN (skip artifact tests)"
fi
Expected:
- Full artifact lifecycle works end-to-end: upload URL, direct upload, list, get, download URL, download, and delete
- downloaded artifact content matches the original payload
- deleted artifact no longer appears in the agent artifact list
Pass/fail:
- Pass if upload succeeds,
DOWNLOADED_ARTIFACT_PAYLOADmatchesARTIFACT_PAYLOAD, andREMAINING_ARTIFACTS=0 - Fail if any artifact step returns non-200, MinIO is up but the lifecycle breaks, or downloaded content mismatches
Phase 10: Durable HITL interrupt persistence
HITL_PERSIST_SID="$(send_message "$HITL_ID" "Request approval for the durable interrupt persistence stress test, then emit stress.hitl.complete.")"
# Wait for interrupt to appear
PERSISTED_INTERRUPT="$(wait_for_interrupt "$HITL_ID" "$HITL_PERSIST_SID")"
echo "$PERSISTED_INTERRUPT" | jq .
PERSISTED_INTERRUPT_ID="$(echo "$PERSISTED_INTERRUPT" | jq -r '.id // empty')"
PERSISTED_SESSION_ID="$(echo "$PERSISTED_INTERRUPT" | jq -r '.sessionId // empty')"
PERSISTED_TOOL_NAME="$(echo "$PERSISTED_INTERRUPT" | jq -r '.toolName // empty')"
PERSISTED_ARGS="$(echo "$PERSISTED_INTERRUPT" | jq -c '.args // empty')"
resolve_interrupt "$HITL_ID" "$HITL_PERSIST_SID" "approve" | jq .
sleep 3
PENDING_AFTER_RESOLUTION="$(curl -s "$API_BASE/api/agents/$HITL_ID/interrupts" \
| jq --arg sid "$HITL_PERSIST_SID" '[.interrupts[]? | select(.sessionId == $sid)] | length')"
Expected:
- interrupt appears in
GET /api/agents/$HITL_ID/interruptswith durable fields - pending interrupt response includes id, sessionId, toolName, and args
POST /api/agents/$HITL_ID/interruptresolves the interrupt- resolved interrupt no longer appears in the pending interrupts list
Pass/fail:
- Pass if
PERSISTED_INTERRUPT_ID,PERSISTED_SESSION_ID,PERSISTED_TOOL_NAME, andPERSISTED_ARGSare present before resolution andPENDING_AFTER_RESOLUTION=0 - Fail if the interrupt is not persisted, the resolve call fails, or the interrupt remains pending after resolution
Post-run cleanup (manual, after human verification)
Do NOT run cleanup as part of the test. Cleanup deletes agents and cascade-deletes their events, sessions, and messages — making post-hoc verification impossible.
Run only after the human has reviewed all results:
npx tsx tests/stress/seed.ts --clean
04 同仓库更多 Skills
stress-test
Full stress-test suite for stablehand — exercises tools, chaining, HITL, multi-model, code execution
smoke-test
Quick stress-test smoke validation for stablehand — verifies each harness and model works
code-review
Provides code review guidelines and best practices
stablehand-cli
Use when operating, debugging, or inspecting a stablehand app via its HTTP API. Wraps agent/turn/conversation/sandbox-session/event/artifact/file/secret/setting/trigger/webhook operations, feedback submission, SSE and AG-UI wire streaming, multipart uploads, and gzip uploads with bundled Bun TypeScript CLIs.
smoke-test
Quick stress-test smoke validation for stablehand — verifies each harness and model works