triaging-test-failures
Triage the failures in a Qase test run: cluster them by error signature, separate real product bugs from automation, environment, and flaky noise, and file defects. Use when the user asks to triage a run; why a run, a build, or the tests in it failed; why last night's or the latest run failed or is red; what broke; to investigate, group, cluster, or classify failures; which failures are real bugs versus noise; or to file defects for a failed run.
- License 许可
- MIT
- 最近更新
- 2026-11-04
- Stars
- ★ 7
- Forks
- 0
01 前言 Front Matter
- name
- triaging-test-failures
- description
- Triage the failures in a Qase test run: cluster them by error signature, separate real product bugs from automation, environment, and flaky noise, and file defects. Use when the user asks to triage a run; why a run, a build, or the tests in it failed; why last night's or the latest run failed or is red; what broke; to investigate, group, cluster, or classify failures; which failures are real bugs versus noise; or to file defects for a failed run.
03 SKILL.md 正文
Quality Supervisor — Failure Triage / Root-Cause Analysis
Turn a run full of red into a short list of causes: grouped failures, a probable cause per group, a bug-versus-noise call, and defects for the ones that are real.
Read the plugin's references/qql.md before writing any query — it sits two
levels up from this skill's directory (../../references/qql.md). Field names
differ per entity and a wrong name is a hard error, not an empty result.
When to use
- "Triage run 512." / "Why are these tests failing?"
- "Which failures are real bugs and which are noise?"
- "File defects for last night's failures."
Prerequisites
- A project code. A run ID if the user has one; otherwise find the latest.
- QQL access (Teams/Enterprise) for the run lookup and history checks. The row-level work goes through the REST API and needs no QQL.
Tools
qase_project_context— project metadata. Call first.qql_search— find the run, check per-case history.qase_api— fetch the run's failing results (see step 3: QQL cannot do this).qase_get— detail on a specific case when writing up a cause.qase_triage_defect— create a defect (write step, needs approval). Read the warning in step 6 before using it.
Workflow
1. Find the run and its real failure count — one query
entity = "run" and project = "CODE" and status = "failed" ORDER BY started DESC
Runs have no created field — started and ended are the only
timestamps, so this is how "the latest run" is expressed.
The response carries both run_id and a stats object with per-status counts.
That stats block is your authoritative failure count. Take the totals from
it and never recompute them by counting rows you fetched — rows come in pages,
stats does not.
If the user named a run, query it by id instead: entity = "run" and project = "CODE" and id = 512.
2. Frame the scale before looking at details
From stats: total, failed, blocked, skipped, invalid, untested. State these up
front. A run with 3 failures out of 5 and one with 3 out of 4,000 are different
problems, and the ratio changes what triage is worth doing.
Note untested separately — those tests never ran, which is a gap, not a
failure, and it is easy to misread as green.
3. Fetch the failing results — via REST, not QQL
QQL cannot scope results to a specific run. The result entity has no
run-ID field; run matches the run title, and titles repeat (autotest runs
are routinely named by timestamp or branch). Filtering by title will silently
mix in other runs' results.
Use qase_api instead:
GET /v1/result/{code}?run={run_id}&status=failed&limit=100
Read filtered from the response for the count matching your filter —
total is the project-wide count and ignores the filter entirely. Cross-check
filtered against stats.failed from step 1; if they disagree, say so rather
than picking one.
If filtered exceeds 100, paginate with offset until you have them all, or
state how many you examined out of how many exist. Add &status=blocked as a
second pass if the run has blocked results worth triaging.
Each row gives you case_id, status, stacktrace, comment, steps,
time_spent_ms, and hash. The stacktrace is what makes clustering possible
and it is only available on these rows — it is not a queryable field, so
there is no way to group by it server-side.
Note: results carry no environment or configuration. If environment matters,
get it from the run (run.environment), which applies to the whole run.
4. Cluster by cause, not by symptom
Group the failures you fetched:
- Same error signature — normalise the stacktrace first (strip line numbers, paths, timestamps, IDs), then group by exception type plus the first meaningful frame. This is the strongest signal.
- Same suite or feature area — points at one broken area.
- Same fixture, setup, or shared step — points at test infrastructure.
- All failures in the run — a single infra or auth failure often explains every red; check this before triaging 200 items individually.
Report the cluster count next to the failure count. Collapsing 40 failures into 3 causes is the useful output; a list of 40 is not.
5. Classify each cluster
One label per cluster, with the evidence that supports it:
- Product bug — deterministic, asserts on real behaviour → defect candidate.
- Automation issue — selector drift, timing, bad assertion, stale test data → fix the test.
- Environment/infra — auth, network, service unavailable, whole-run failure → fix the environment.
- Flaky — the same case has passed and failed recently → hand to
analyzing-test-flakiness; do not file a product bug.
Check history per case before calling anything flaky:
entity = "result" and project = "CODE" and caseId = 234 ORDER BY ended DESC
Note caseId (not case, which is the title) and ended (results have no
created). Mixed pass/fail history means flaky. Failures only, never a pass,
means the opposite — that is a real, persistent failure, so treat it as a bug
candidate rather than noise.
6. File defects — only after approval, and only with honest claims
Before creating anything, search for an existing defect:
entity = "defect" and project = "CODE" and status = "Open"
Match on title and error signature; if one already covers the cause, reference it instead of creating a duplicate.
Then, per confirmed product-bug cluster, one defect per cause — not one per failing test.
qase_triage_defect requires title, actual_result, and severity — all
three, since the API rejects a defect without them. Severity is one of
blocker, critical, major, normal, minor, trivial, undefined.
The defect cannot be linked to the failing results. Qase's API has no
defect-side way to attach them: the results and runs fields on a defect are
populated from the runner side, when a result is submitted already marked as a
defect. Nothing this plugin can call will create that link after the fact.
So make the defect self-sufficient. Put the run ID, the affected case IDs, the
normalised error signature, and the environment into actual_result and
description — that text is the only durable connection between the defect and
its evidence, so it has to carry all of it.
Report the defect IDs you created, and don't imply the failing results were attached to them.
Output format
## Triage — <PROJECT> run <ID> (<run title>)
_Generated by Qase Quality Supervisor · triaging-test-failures skill_
Run stats: <total> tests · <failed> failed · <blocked> · <skipped> · <untested> untested
Examined: <N> failing results (of <filtered>) | Clusters: <K>
### Cluster 1 — <normalised error signature>
- Cases: <ids> · Occurrences: <n> · Environment: <run environment, if any>
- Classification: <Product bug | Automation | Environment | Flaky> (confidence: H/M/L)
- Probable cause: <one or two lines, tied to the evidence>
- Action: <file defect | fix test | fix env | route to analyzing-test-flakiness>
### Summary
- Product bugs: <n> · Automation: <n> · Environment: <n> · Flaky: <n>
- Untested in this run: <n> — never executed, not failures
### Recommended actions (each needs your approval)
- <e.g. "File one defect for the auth-timeout cluster covering C-12, C-14, C-19">
### Evidence
<the queries and API calls used, verbatim>
---
_Generated by Qase Quality Supervisor · triaging-test-failures skill_
Guardrails
- Read and analyse first; create defects only after the user approves.
- One defect per root cause, and search for an existing open defect first.
- Never call a
*_deletetool. (A hook blocks them anyway.) - Don't over-claim causation — give a confidence level and cite the evidence.
- Take counts from run
stats, not from the rows you happened to fetch; if you examined a subset, say so. - Never claim results were linked to a defect — the API cannot do it.
- Suspected flaky goes to
analyzing-test-flakiness, not into a product bug.
04 同仓库更多 Skills
reporting-quality-pulse
Build a visual "Quality Pulse" report card for a Qase project over any period the user chooses — daily, weekly, biweekly, monthly, sprint, or a custom date range. A styled HTML card summarizing test execution, pass rate, defect backlog, case activity, and active users for that window. Use this whenever the user asks for a quality pulse, QA pulse, quality/test report card, test health summary, sprint QA snapshot, release health check, or any "how is testing going" overview for a Qase project over some period, even if they don't say the word "pulse".
linking-test-cases
Link a repository's autotests to Qase test cases by meaning: propose a mapping for confirmation, write the confirmed case IDs into the test code in each Qase reporter's own syntax, and mark the linked cases Automated. Use when the user asks to link, map, match, or connect autotests or automated tests to Qase cases; to add Qase IDs, QaseId annotations, or case IDs to tests; why tests don't match their cases in Qase; or why automated results land in Qase as new cases instead of the existing ones.
assessing-release-readiness
Produce an evidence-backed go / no-go assessment for a Qase release scope — a milestone, a test plan, or a set of runs. Use when the user asks whether they are ready to ship, release, or cut a version; for a go/no-go, a release sign-off, or a quality gate; what is blocking the release or holding it up; whether a milestone or sprint is done; how a release is tracking; or whether it is safe to deploy.
analyzing-test-flakiness
Find genuinely flaky tests in a Qase project — cases that both pass and fail — and separate them from tests that are simply broken. Use when the user asks which tests are flaky, unreliable, unstable, intermittent, or non-deterministic; about test stability, flakiness, or the flake rate; which tests pass sometimes and fail other times, or fail randomly; which tests to quarantine, mute, or flag as flaky; or whether a specific test is flaky or a real regression.
analyzing-test-coverage
Find where a Qase project's test coverage is thin. Use when the user asks about test coverage or coverage gaps; what isn't tested, untested areas, or what is missing tests; which cases are still manual, not automated, or yet to be automated; which cases have never been run or executed; which suites are empty, thin, or hold no cases; how much of the project is automated; which cases are stale or outdated; or wants test cases drafted to close a gap.
analyzing-change-impact
Analyze what a feature branch changes and produce a QA-facing impact analysis plus a regression/new checklist, grounded in the project's existing Qase cases. Use when the user asks for an impact analysis, change impact, risk analysis, or blast radius of a branch or PR; what QA should test for these changes; a QA checklist or test plan for a branch; what could break; or which existing tests to re-run before merging. Reads the repository diff and Qase; renders an HTML report.