evaluation-and-logs
Inspect AlpaSim ASL rollouts, extract camera streams, evaluate and aggregate metrics, re-evaluate completed runs, render diagnostic videos, and analyze trajectories, maps, and geometry without producing new simulations.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- evaluation-and-logs
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Inspect AlpaSim ASL rollouts, extract camera streams, evaluate and aggregate metrics, re-evaluate completed runs, render diagnostic videos, and analyze trajectories, maps, and geometry without producing new simulations.
- disable-model-invocation
- true
02 资源文件 8
- 参考文档references/asl-format.md3.8 KB
- 参考文档references/evaluation-workflows.md5.0 KB
- 参考文档references/geometry-and-utilities.md4.0 KB
- 参考文档references/metrics-and-videos.md5.2 KB
- 参考文档references/troubleshooting.md6.3 KB
- 脚本scripts/asl_to_frames.py8.7 KB
- 脚本scripts/check_run_metadata.py5.1 KB
- 脚本scripts/print_asl.py3.5 KB
03 SKILL.md 正文
Evaluation and logs
Use this sub-skill when the task starts with an existing AlpaSim run, .asl log,
rollout directory, metrics parquet, evaluation configuration, trajectory, map,
or diagnostic video. It is for reading, scoring, aggregating, and explaining
outputs—not for starting the simulator or fixing controller, physics, traffic,
renderer, or deployment services.
Route first
- Runtime production, replay service orchestration, daemon health, or timing
failures: route to
runtime-services. - Wizard setup, scene/model acquisition, Docker/Slurm deployment, or run
creation: route to
simulation-wizard. - Driver policy/model implementation: route to
drivers-and-plugins. - Controller, physics, or traffic implementation: route to
control-physics-traffic. - Proto compilation, generated stubs, or contributor tooling: route to
grpc-and-developer-tools.
Keep this sub-skill for the evidence and post-run analysis boundary.
Choose the smallest workflow
- Identify the run root and whether it is a single job or an array-job parent.
A single job has
eval-config.yamlat its root; an array parent has child job directories containing that file. Confirmwizard-config.yamlis present before using automatic re-evaluation. - Locate complete rollouts under
rollouts/<scene-or-clip-id>/<rollout-id>/. Treat a rollout as complete only when its_completemarker exists. Expectrollout.asl, optionalmetrics.parquet, and optional.mp4files there. - For a quick log diagnosis, read the ASL format reference
and use the bundled
scripts/print_asl.py. Start with--just-types, a narrow--start/--endrange, and selected--message-types; payloads containing images or video-model bytes are redacted before printing. Use--strictwhen truncation must fail loudly. - For camera inspection, use
scripts/asl_to_frames.pywith a quoted recursive glob. Choose--format framesfor bounded file inspection or--format mp4when ffmpeg/imageio is available. Keep--max-filesand--max-concurrencybounded for large run trees. - For post-evaluation, run the documented module form in evaluation workflows. Read geometry and utilities before comparing trajectories or maps. Evaluation requires the resolved YAML, complete ASLs, and a matching USDZ glob when map-backed metrics or videos are enabled. It writes per-rollout parquet beside each ASL.
- Aggregate only after all intended jobs have metrics. The aggregation step
discovers
rollouts/**/metrics.parquet, writes the aggregate tree, and may create relative video links. For a prior run, preferpython -m eval.reevalor the equivalent installed re-evaluation command; never submit scheduler work from this sub-skill without an explicit user request.
Read the data correctly
- ASL is a big-endian, four-byte size-delimited protobuf stream of
LogEntrymessages. The first meaningful message should berollout_metadata; actor poses are timestamped and includeEGO. Requests/returns, camera images, routes, and traffic predictions are additional evidence. - Use timestamps (
timestamp_us) rather than list positions when correlating poses, plans, camera frames, and metrics.async_read_pb_log()is tolerant by default: a short final frame logs a warning and stops. Setraise_on_malformed=Truefor diagnosis or validation. - Evaluation loads ASL through
load_scenario_eval_input_from_asl(), thenScenarioEvaluator(EvalConfig).evaluate(...)returns per-timestepMetricReturnvalues, aggregated values, and a Polars metrics dataframe. Missing rollout metadata, actor definitions, ground truth, or required response/camera data can make a metric unavailable or the input invalid. - Map-dependent offroad and video analysis need a discoverable USDZ artifact
and compatible map data. Without a vector map, aggregation supplies an
offroad=0.0fallback and records a warning; do not interpret that as proof that offroad was measured.
Interpret outputs
Use metrics and videos for the complete
metric and output contract. In short: per-rollout metrics live next to the ASL;
aggregate/metrics_results.txt, metrics_results.parquet,
metrics_unprocessed.parquet, results-summary.json, and
metrics_results.png summarize a run. Videos are named from clip, rollout,
camera, and layout identifiers and are organized under aggregate/videos/.
Inspect the raw parquet and the modifiers recorded in the text summary before
comparing runs.
Programmatic analysis
- Use
alpasim_utils.logs.read_trajectory()for a scene name plus the EGO trajectory extracted from actor-pose messages. It returnsNonewhen there are no usable actor poses or no metadata scene id. - Use
alpasim_utils.geometryfor gRPC conversion,Pose,Polyline, and timestampedTrajectory. Internally, quaternion arrays are(x, y, z, w); gRPC fields are addressed as(w, x, y, z). Preserve the rig-to-AABB-center transform from rollout metadata before comparing map or vehicle geometry. - Use
Trajectory.interpolate_pose()only inside its time range (a one-pose trajectory accepts its exact timestamp); sort or validate timestamps before constructing trajectories. Use Shapely objects for 2-D intersections and distances, and state when z is intentionally ignored. - Treat the Rust-backed
utils_rstypes as an optional native boundary: verify that the extension and generated protobufs import before relying on geometry helpers. The Python evaluation path is not a substitute for a missing map, renderer, ffmpeg, or model asset.
Safe checks and failure recovery
Run python -m ... --help, import checks, and bounded fixture checks before a
large evaluation. Do not run the original checkout's setup, asset download,
Docker build, Slurm submission, or credentialed model acquisition from here.
Use troubleshooting for install/import,
optional backend, data/config, CLI/API, and workflow failures. When evidence
is incomplete, report the exact missing message, asset, marker, config field,
or backend rather than filling in a metric or claiming a successful run.
Bundled helpers
scripts/print_asl.py: bounded, redacted protobuf inspection; safe and read-only.scripts/asl_to_frames.py: bounded extraction to images or MP4; writes only under the requested output directory and never downloads data.scripts/check_run_metadata.py: read-only single/array-job layout and ASL metadata checker; it does not load USDZ maps or rewrite configs.
The package's print-asl and asl-to-frames console launchers currently
request a main symbol that the installed modules do not define. Use
python -m alpasim_utils.print_asl and python -m alpasim_utils.asl_to_frames
(or these bundled helpers) until packaging is corrected. The evaluation,
aggregation, and re-evaluation module entry points expose main and their
python -m forms are the portable fallback.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).