compute-model-assessor
Assess task complexity and recommend optimal model tiers and thinking budgets (High, Medium, Low, Off) calibrated for Gemini, Claude, and OpenAI models. Do not use for mechanical syntax fixes or non-LLM triage.
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- compute-model-assessor
- description
- Assess task complexity and recommend optimal model tiers and thinking budgets (High, Medium, Low, Off) calibrated for Gemini, Claude, and OpenAI models. Do not use for mechanical syntax fixes or non-LLM triage.
02 资源文件 2
- 参考文档CARD.md6.8 KB
- 参考文档config.default.yaml298 B
03 SKILL.md 正文
Compute & Model Assessor Engine
compute-model-assessor is the authoritative task classification, dimensional scoring, token economics, reasoning budget allocation, and dynamic trajectory escalation engine for Brain Harness. Backed by harness.services.compute_assessor, it evaluates incoming user prompts, architectural tasks, and multi-agent subtasks across 5 orthogonal dimensions (Ambiguity, Span, Depth, Rigor, Concurrency)—recommending the optimal model tier and thinking budget (High, Medium, Low, Off) to maximize solution quality while eliminating latency and token waste.
Every compute assessment follows a strict five-stage progression:
[1. Task Surface Analysis] → [2. Compute Calibration & Economics] → [3. Visual Budget Brief] → [4. Recommendation Checkpoint] → [5. Routing Commit & Escalation]
See CARD.md for the quick-reference summary card, calibration heuristics matrix, and invariants checklist.
Consult /crafting-skills for skill design standards and /deepen-architecture for architectural assessment.
1. Task Surface Analysis & Scoring Profiles
Inspect the incoming user task across 5 core complexity dimensions using DimensionalScorer and configurable ScoringProfile presets:
-
Solution Ambiguity & Branching (
ambiguity):- Does the prompt require exploring multiple alternative designs or reconciling conflicting constraints?
- High (0.7–1.0): Novel architecture design, ambiguous bug diagnosis, race condition debugging.
- Medium (0.4–0.6): Implementing well-specified components, writing unit tests, refactoring with fixed contracts.
- Low (0.1–0.3): Mechanical syntax edits, regex drafting, boilerplate generation, formatting.
-
Context Span & Codebase Depth (
span):- How many files, interfaces, or dependency layers are impacted?
- Multi-module / DAG: Requires cross-file reasoning and architectural invariant preservation.
- Single-seam: Confined to one cohesive module or file.
- Zero-context: Self-contained prompt with no workspace dependency.
-
Algorithmic Depth & AST Manipulation (
depth):- Does the task involve type checking, parser construction, topological sorting, or complex algorithms?
-
Execution Rigor & Verification Cost (
rigor):- What is the penalty for a hallucinated or flawed first attempt?
- Critical: Structural migrations, kernel changes, persistent state mutations.
- Standard: Routine feature additions with immediate automated test verification.
- Trivial: Throwaway scratch scripts or informational answers.
-
Concurrency & Race Potential (
concurrency):- Does the task involve async event loops, thread safety, locks, semaphores, or distributed synchronization?
Scoring Profile Presets:
balanced(default): General balanced weights across all 5 dimensions.reasoning_heavy: Lower threshold for high reasoning, elevated weight on ambiguity and depth.cost_optimized: Higher threshold for high reasoning, minimizes token expenditure.latency_optimized: Biased towards faster turnaround tiers.
Completion criterion: Task scored across the 5 dimensions, producing a
ComplexityVectorand anAssessmentTrace.
2. Compute Calibration, Economics & Multi-Provider Matrix
Map the assessed complexity level to explicit model tiers, token economics projections, and provider-specific reasoning configurations:
┌─────────────────────────────────────────────────────────────────────────┐
│ CALIBRATION HEURISTICS MATRIX │
├────────────┬─────────────────────┬───────────────────┬──────────────────┤
│ Complexity │ Gemini 3.7 Flash │ Claude 3.7 Sonnet │ OpenAI o-series │
├────────────┼─────────────────────┼───────────────────┼──────────────────┤
│ High │ Thinking: `High` │ Budget > 16k tok │ o3-mini (high) │
├────────────┼─────────────────────┼───────────────────┼──────────────────┤
│ Medium │ Thinking: `Medium` │ Budget ~4k–8k tok │ o3-mini (medium) │
├────────────┼─────────────────────┼───────────────────┼──────────────────┤
│ Low / Off │ Thinking: `Low/Off` │ Standard / Off │ gpt-4o-mini / Off│
└────────────┴─────────────────────┴───────────────────┴──────────────────┘
- High Complexity (Deep Reasoning Tier):
- Primary: Gemini 3.7 Flash (
thinking_budget="high", 16,384 tokens), Claude 3.7 Sonnet (budget_tokens=16000),o3-mini(effort="high"). - Alternative / Open: DeepSeek-R1 (Full reasoning), Qwen-2.5-Coder-32B.
- Primary: Gemini 3.7 Flash (
- Medium Complexity (Balanced Agentic Tier):
- Primary: Gemini 3.7 Flash (
thinking_budget="medium", 4,096 tokens), GPT-4o, Claude 3.5 Sonnet. - Alternative / Open: DeepSeek-V3, Qwen-2.5-Coder-14B.
- Primary: Gemini 3.7 Flash (
- Low / Mechanical Complexity (High-Throughput Tier):
- Primary: Gemini 2.0 Flash / Flash-Lite (
thinking_budget="off", 0 tokens), GPT-4o-mini, Claude 3.5 Haiku. - Alternative / Open: Llama-3.3-8B-Instruct, Mistral-Small.
- Primary: Gemini 2.0 Flash / Flash-Lite (
Token Economics Projection (ComputeEconomics):
estimated_query_cost_usd: Projected USD cost per query.expected_latency_p50_seconds/expected_latency_p95_seconds: Latency percentile distributions.
Completion criterion: Concrete model recommendation, token economics, and thinking budget declared with provider-specific configuration arguments.
3. The Visual Budget Brief & Live Provider Studio
Synthesize complexity evaluation, model tiers, and cost/latency tradeoffs into an interactive HTML visual brief with live multi-provider payload copying using ComputeVisualBriefGenerator:
- Programmatic Generation:
from harness.services.compute_assessor import ComputeRouter html_path = ComputeRouter.generate_visual_brief( "Refactor kernel service registry", files_count=4, is_architecture=True, profile="balanced", ) - Target Location: Writes to
%TEMP%\compute-assessor-<timestamp>.html(Windows) or/tmp/compute-assessor-<timestamp>.html(Unix). - Visual Standards & Live Studio:
- Dark mode (
#0d1117) with Tailwind CSS and Mermaid.js. - 5-Dimensional Complexity Vector bars, Composite Score badge, and Token Economics card.
- Interactive Mermaid Decision DAG.
- Live Provider Payload Studio: Interactive tabs to preview and copy exact JSON payloads for Google Gemini, Anthropic Claude, OpenAI o-series, DeepSeek, and Ollama.
- Dark mode (
- Delivery: Surface the absolute file path with clickable links to the user.
Completion criterion: Self-contained HTML report written to
%TEMP%and delivered to user.
4. Recommendation Checkpoint & CLI Integration
Present the compute recommendation and pause for user or harness confirmation:
CLI Command
# Markdown block output with profile
harness assess-compute "Refactor kernel service registry" --arch --files 4 --profile balanced
# JSON output with economics
harness assess-compute "Fix typo in docstrings" --json
# Generate HTML visual brief
harness assess-compute "Migrate persistent database" --html
Formal Structured Compute Recommendation Block:
### [Compute Recommendation Block]
- **Complexity Assessment**: `High` (Tier: `high_reasoning`)
- **Primary Recommendation**: `gemini-3.7-flash` (Thinking Level: `HIGH`)
- **Reasoning Budget Tokens**: `~16,384 tokens`
- **Alternative / Peer Models**: `claude-3-7-sonnet | o3-mini | deepseek-r1`
- **Rationale**: High complexity: cross-module scope, structural ambiguity, or architectural constraints.
- **Vector Breakdown**: Ambiguity: 0.6, Span: 0.9, Depth: 0.8, Rigor: 0.7, Concurrency: 0.1
- **Economics Projection**: ~$0.0105 est. cost | Latency: p50 ~4.5s, p95 ~12.0s
Completion criterion: Structured recommendation block emitted and approved.
5. Routing Commit, Dynamic Escalation & EventBus Telemetry
Commit the assessed configuration directly to the agent runtime, handle reactive retry escalation, and emit immutable audit events to the kernel EventBus:
IoC Service & EventBus Audit:
from harness.services.compute_assessor import (
COMPUTE_ASSESSOR_SERVICE,
ComputeAssessorService,
DynamicTrajectoryEscalator,
TrajectoryState,
)
# IoC Container resolution
assessor_service = ctx.require(COMPUTE_ASSESSOR_SERVICE)
# Asynchronous assessment with append-only EventBus telemetry
assessment = await assessor_service.assess_and_publish("Refactor database schema", files_count=3, is_architecture=True)
payload = assessor_service.synthesize_payload(assessment)
Dynamic Trajectory Escalation on Error / Retry:
# Track agent trajectory across attempts
trajectory = TrajectoryState()
trajectory.record_attempt(success=False, error="Syntax error during compilation")
# Automatically escalate thinking budget (e.g. Fast -> Standard -> High Reasoning)
escalated_assessment = assessor_service.escalate(assessment, trajectory)
print(f"Escalated model tier: {escalated_assessment.model_tier} (Budget: {escalated_assessment.budget_tokens:,} tokens)")
Hierarchical Tree Budget Allocation:
# Proportional token allocation across swarm tree branches
allocated_shares = DynamicTrajectoryEscalator.allocate_tree_budget(
total_budget_tokens=32000,
branch_weights=[0.5, 0.25, 0.25],
)
# Returns [16000, 8000, 8000]
Completion criterion: Target execution context configured with recommended model, thinking parameters, and trajectory escalation policy.
Anti-Patterns
- Meta-Latency Waste — Running an expensive high-reasoning model call just to decide if a simple task needs a high-reasoning model. Use rule-based heuristics or low-latency classification.
- Context Blindness — Underestimating the reasoning requirements of multi-file architectural refactors and selecting low-budget tiers.
- Speculative Over-Reasoning — Burning maximum thinking budget on mechanical formatting, boilerplate, or simple regex generation.
- Lock-in Rigidity — Hardcoding model names without providing cross-provider equivalents for Claude, OpenAI, and open-weights models.
- Bypassing User Overrides — Forcing automated routing without allowing explicit user override of model tier.
04 同仓库更多 Skills
ui-behavior-primitive-architect
Architect, audit, and verify zero-dependency headless UI behavior primitives, WebMCP agent bridges, WAI-ARIA/SSR/RTL tri-modal checks, and framework expert routing. Do not use for static CSS theme styling or backend database migrations.
trustgraph-ontology-governor
Govern knowledge graphs, SHACL-AF triple policies, OntologyRAG subclass validation, PROV-O dual named graphs, and Laya entropy decision scoring. Do not use for relational SQL queries or generic web scraping.
shim-harness-governor
Govern universal CLI agent loops, 4-tier tool permissions, ancestor-symlink workspace sandboxes, deterministic-first command routing, collision-safe MCP namespacing, and layered skill loading. Do not use for static UI styling or external SQL analytics.
graphiti-zep-memory-governor
Govern bi-temporal knowledge graphs with Graphiti future-edge clamping, MinHash/LSH entity dedup, RRF/MMR reranking, and Zep typed ontologies. Do not use for static SQL joins or generic web crawling.
agora-mcp-workbench-architect
Architect, audit, and orchestrate Agora Workbench MCP servers, loopback sidecars, unified state graphs, gateway/dispatcher connectors, and policy-enforced data lakes. Do not use for static frontend CSS styling or non-MCP desktop GUI automation.
skill-eval-assurance-runner
Evaluate and assure agent skills using small-N Wilson confidence intervals, unbiased pass^k reliability, deterministic trajectory graders, preflight boundary honesty, and sealed digest verification. Do not use for subjective prose review or unconstrained production deployment.