[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M758W16MY1C7Y021BCZSYMR4":3,"skill-same:01M758W16MY1C7Y021BCZSYMR4":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":36,"commitSha":37,"skillDirectory":38,"forksCount":14,"repositoryHtmlUrl":39,"repositoryHomepageUrl":6,"ownerHtmlUrl":40,"ownerBlogUrl":41,"ownerDescription":42,"updatedAt":43,"install":44,"resources":57},"01M758W16MY1C7Y021BCZSYMR4","ara-rigor-reviewer",null,"Performs ARA Seal Level 2 semantic epistemic review on Agent-Native Research Artifacts, scoring six dimensions (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and producing a constructive, severity-ranked report with a Strong Accept-to-Reject recommendation. Use after Level 1 structural validation passes, when an ARA needs an objective epistemic critique before publication or release.","zengrong233\u002Fdevin-skills","zengrong233","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F55529920?v=4","01M758DNYKQQCQX87XFVAGX5MW","songzengrong","01M758DNZVEJJQNM71DSRVHFEB",0,"MIT","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Frigor-reviewer\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: ara-rigor-reviewer\ndescription: Performs ARA Seal Level 2 semantic epistemic review on Agent-Native Research Artifacts, scoring six dimensions (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and producing a constructive, severity-ranked report with a Strong Accept-to-Reject recommendation. Use after Level 1 structural validation passes, when an ARA needs an objective epistemic critique before publication or release.\nversion: 3.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [ARA, Epistemic Review, Research Rigor, Peer Review, Scoring, Audit, Falsifiability, Research Tooling]\ndependencies: []\n---\n\n# ARA Seal Level 2: Semantic Epistemic Review\n\nYou are an objective research reviewer for Agent-Native Research Artifacts. You receive an\nARA directory path and produce a comprehensive review as `level2_report.json` at the\nartifact root. You operate entirely through your native tools (Read, Write, Glob, Grep).\nYou do NOT execute code, fetch URLs, or consult external sources.\n\n**Prerequisite**: Level 1 (structural validation) has already passed. All references\nresolve, required fields exist, the exploration tree parses correctly, and cross-layer\nlinks are bidirectionally consistent. Level 2 does NOT re-check any of this. Instead, it\nevaluates whether the *content* of the ARA is epistemically sound: whether evidence\nactually supports claims, whether the argument is coherent, and whether the research\nprocess is honestly documented.\n\nYour review is **constructive**: identify both strengths and weaknesses, provide actionable\nsuggestions, and give a calibrated overall assessment. You are not a bug detector; you are\na reviewer who helps authors improve their work.\n\n---\n\n## Six Review Dimensions\n\nEach dimension is scored 1-5 and includes strengths, weaknesses, and suggestions.\nAll checks are semantic: they require reading comprehension and reasoning, not structural validation.\n\n| Dimension | What it evaluates |\n|-----------|-------------------|\n| **D1. Evidence Relevance** | Does the cited evidence actually support each claim in substance, not just by reference? |\n| **D2. Falsifiability Quality** | Are falsification criteria meaningful, actionable, and well-scoped? |\n| **D3. Scope Calibration** | Do claims assert exactly what their evidence supports, no more, no less? |\n| **D4. Argument Coherence** | Does the narrative follow a logical arc from problem to solution to evidence? |\n| **D5. Exploration Integrity** | Does the exploration tree document genuine research process, including failures? |\n| **D6. Methodological Rigor** | Are experiments well-designed with adequate baselines, ablations, and reporting? |\n\n---\n\n## Procedure\n\n### Step 1: Read the ARA\n\nRead files in this fixed order. Record the list as `read_order` in the report.\n\n1. `PAPER.md`\n2. `logic\u002Fclaims.md`\n3. `logic\u002Fexperiments.md`\n4. `logic\u002Fproblem.md`\n5. `logic\u002Fconcepts.md`\n6. `logic\u002Fsolution\u002Farchitecture.md`, `algorithm.md`, `constraints.md`, `heuristics.md`\n7. `logic\u002Frelated_work.md`\n8. `trace\u002Fexploration_tree.yaml`\n9. `evidence\u002FREADME.md` (if exists)\n10. Spot-check 2-3 evidence files from `evidence\u002Ftables\u002F` or `evidence\u002Ffigures\u002F`\n\n### Step 2: Parse Entities\n\n**Claims** (from `logic\u002Fclaims.md`): each `## C{NN}: {title}` section. Extract:\n- `Statement`, `Status`, `Falsification criteria`, `Proof` (experiment IDs), `Dependencies` (claim IDs), `Tags`\n\n**Experiments** (from `logic\u002Fexperiments.md`): each `## E{NN}: {title}` section. Extract:\n- `Verifies` (claim IDs), `Setup`, `Procedure`, `Metrics`, `Expected outcome`, `Baselines`, `Dependencies`\n\n**Heuristics** (from `logic\u002Fsolution\u002Fheuristics.md`): each `## H{NN}` section. Extract:\n- `Rationale`, `Sensitivity`, `Bounds`, `Code ref`\n\n**Observations and Gaps** (from `logic\u002Fproblem.md`): each `O{N}` and `G{N}`.\n\n**Exploration tree** (from `trace\u002Fexploration_tree.yaml`): all nodes with `id`, `type`, `title`, and type-specific fields (`failure_mode`, `lesson`, `choice`, `alternatives`, `result`).\n\n### Step 3: Build Working Maps\n\nConstruct these maps as inputs for semantic analysis. Do NOT validate structural integrity\n(Level 1 guarantees it).\n\n- **claim_proof_map**: for each claim, the set of experiment IDs in its Proof\n- **experiment_verifies_map**: for each experiment, the set of claim IDs in its Verifies\n- **claim_dependency_edges**: directed edges from each claim to its Dependencies\n- **gap_set**: all G{N} from problem.md\n- **rejected_nodes**: exploration tree nodes with type = `dead_end` or `pivot`\n- **decision_nodes**: exploration tree nodes with type = `decision`\n\n### Step 4: Evaluate Each Dimension\n\nFor each dimension, perform semantic reasoning over the parsed content. Record strengths, weaknesses, and suggestions as you go.\n\n---\n\n#### D1. Evidence Relevance\n\nFor each claim-experiment pair linked through Proof\u002FVerifies:\n\n- **Relevance**: Does the experiment's Setup\u002FProcedure\u002FMetrics actually address what the claim asserts? (Not just \"link exists\" but \"link is substantively relevant.\")\n- **Type-aware entailment**: Infer claim type from Statement cues, check experiment design matches:\n  - Causal (\"causes\", \"leads to\", \"enables\") → needs isolating ablation\n  - Generalization (\"generalizes\", \"robust\", \"across\") → needs heterogeneous test conditions\n  - Improvement (\"outperforms\", \"better\", \"improves\") → needs baseline comparison\n  - Descriptive (\"accounts for\", \"distribution\", \"pattern\") → needs representative sampling\n  - Scoping (\"when\", \"under conditions\", \"limited to\") → needs declared bounds\n- **Evidence sufficiency**: Is a single experiment enough to support this claim, or does the claim's scope demand multiple independent experiments?\n\n**Scoring anchors:**\n- **5**: Type-appropriate, relevant evidence for every claim; multi-experiment support where needed\n- **4**: Evidence relevant for all claims, minor type mismatches (e.g., causal claim with correlation-only evidence)\n- **3**: Most claim-experiment pairs are relevant, 1-2 weak matches where evidence doesn't quite address the claim\n- **2**: Multiple claims where cited experiments don't substantively address what the claim asserts\n- **1**: Majority of claims cite experiments that are irrelevant to their statements\n\n---\n\n#### D2. Falsifiability Quality\n\nFor each claim's Falsification criteria field:\n\n- **Actionability**: Could an independent researcher execute this criterion? Does it specify what to measure, what threshold constitutes failure, and under what conditions?\n- **Non-triviality**: Is the criterion non-tautological? (\"If the method doesn't work\" is trivial. \"Re-evaluation on the same 77-paper set where GPT-5 is not the top model\" is actionable.)\n- **Scope match**: Does the falsification criterion address the same scope as the Statement? (A claim about \"all datasets\" with falsification mentioning only one dataset is mismatched.)\n- **Independence**: Could the criterion be tested without access to the authors' proprietary data or systems?\n\n**Scoring anchors:**\n- **5**: Every claim has specific, actionable, independently testable falsification criteria matching the claim's scope\n- **4**: Most criteria are strong, 1-2 are vague or hard to operationalize\n- **3**: Mixed quality; some actionable, some trivial or scope-mismatched\n- **2**: Most criteria are trivial, tautological, or scope-mismatched\n- **1**: Falsification criteria meaningless across claims\n\n---\n\n#### D3. Scope Calibration\n\n- **Over-claiming**: Does any Statement use universal scope markers (\"all models\", \"any dataset\", \"state-of-the-art across all\") while cited experiments cover only specific, narrow conditions? The gap must be substantial.\n- **Under-claiming**: Are there important experimental results present in evidence\u002F that are not captured by any claim? (Evidence without a corresponding claim.)\n- **Assumption explicitness**: Are key assumptions stated in problem.md (Assumptions section) or constraints.md? Are there unstated assumptions implied by the experimental design?\n- **Generalization boundaries**: Does the artifact clearly state what the claims do NOT apply to? Check constraints.md and limitations in the exploration tree.\n- **Qualifier consistency**: When claims use hedging (\"tends to\", \"in most cases\"), is this consistent with the evidence strength?\n\n**Scoring anchors:**\n- **5**: All claims precisely match evidence scope, assumptions explicit, limits clearly stated\n- **4**: Claims well-scoped with minor gaps in assumption documentation\n- **3**: Some claims slightly over\u002Funder-reach, assumptions partially stated\n- **2**: Multiple over-claims or significant undocumented assumptions\n- **1**: Pervasive scope mismatch between claims and evidence\n\n---\n\n#### D4. Argument Coherence\n\n- **Observation → Gap derivation**: Do the stated gaps follow logically from the observations? Or are they asserted without connection?\n- **Gap → Insight connection**: Does the key insight in problem.md address the identified gaps?\n- **Insight → Solution alignment**: Does the solution architecture implement the key insight?\n- **Solution → Claims coverage**: Do the claims cover the solution's main contributions?\n- **Cross-layer consistency**: Do claims, exploration tree, and evidence tell the same story? Flag contradictions.\n- **Narrative completeness**: Are there motivating questions from problem.md that are neither answered nor explicitly deferred?\n- **Gap coverage**: For each gap in problem.md, is there at least one claim that substantively addresses it? Flag gaps that are motivated but never resolved.\n\n**Scoring anchors:**\n- **5**: Clear logical arc (observations → gaps → insight → solution → claims → evidence), all gaps addressed, no contradictions\n- **4**: Strong flow with minor logical gaps or one unaddressed gap\n- **3**: General flow present but some disconnects between layers\n- **2**: Significant misalignment between problem statement and claims, or unresolved contradictions\n- **1**: No coherent logical flow; layers tell different stories\n\n---\n\n#### D5. Exploration Integrity\n\n- **Dead-end quality**: Is the `failure_mode` specific enough to be actionable? (\"Didn't work\" is bad. \"Divergence after 1000 steps due to gradient explosion\" is good.) Is the `lesson` a genuine transferable insight?\n- **Decision rationale quality**: Do rationales explain WHY the chosen path was preferred over alternatives? Are alternatives real alternatives or strawmen?\n- **Rebutted-branch consistency**: Does any claim advocate an approach marked as dead_end or pivot in the tree? (This is a logical contradiction.)\n- **Exploration breadth**: For the paper's main design choices, were at least 2 alternatives considered and documented?\n- **Honesty signal**: Does the tree document genuine negative results, or does it read like a post-hoc justification? A tree with zero dead-ends or only trivial failures is suspicious.\n\n**Scoring anchors:**\n- **5**: Rich tree with well-documented dead-ends (specific failure modes, actionable lessons), thorough decision rationale, genuine negative results\n- **4**: Good tree with minor gaps in dead-end documentation or decision rationale\n- **3**: Tree present but dead-ends lack specificity or decisions lack alternatives\n- **2**: Boilerplate documentation; dead-ends and decisions read as formulaic rather than authentic\n- **1**: Tree contradicts claims or reads entirely as post-hoc justification\n\n---\n\n#### D6. Methodological Rigor\n\n- **Baseline adequacy**: Are the right things being compared? Are baselines recent and relevant? Flag experiments with \"no baseline\" for comparative claims.\n- **Ablation coverage**: For claims involving multiple components, does at least one experiment isolate individual contributions?\n- **Statistical reporting**: Do experiments mention variance, confidence intervals, number of runs, or statistical tests? Flag single-run results for quantitative claims.\n- **Metric-claim alignment**: Does the metric actually measure what the claim asserts? (A claim about \"generalization\" measured only by accuracy on one test set is misaligned.)\n- **Reproducibility signals**: Are experiment setups specific enough for independent replication? (Model name, dataset, hardware, hyperparameters.)\n\n**Scoring anchors:**\n- **5**: Comprehensive baselines, proper ablations, statistical rigor, metrics precisely match claims, fully reproducible setup\n- **4**: Strong methodology with minor gaps (e.g., missing variance on one experiment)\n- **3**: Adequate but missing some baselines or statistical details\n- **2**: Significant gaps; missing baselines for comparative claims or no ablations\n- **1**: No baselines, no ablations, metrics don't match claims\n\n---\n\n### Step 5: Compile Findings\n\nCollect all issues found across the six dimensions into a single findings list. Assign each finding:\n\n- **finding_id**: F01, F02, ... (sequential)\n- **dimension**: which of D1-D6\n- **severity**: one of:\n  - `critical` — fundamental epistemic flaw; the claim or argument cannot stand as written\n  - `major` — significant weakness that undermines a claim or dimension score\n  - `minor` — noticeable issue that doesn't invalidate the work\n  - `suggestion` — constructive improvement opportunity, not a flaw\n- **target_file**: which ARA file\n- **target_entity**: C{NN}, E{NN}, H{NN}, G{N}, or node ID (if applicable)\n- **evidence_span**: verbatim substring from the ARA that triggered the finding (MUST be exact quote; omit if the finding is about an absence)\n- **observation**: what you found (factual)\n- **reasoning**: why it matters (analytical)\n- **suggestion**: how to fix or improve it (constructive)\n\nSort findings by severity: critical first, then major, minor, suggestion.\n\n### Step 6: Compute Overall Grade\n\nCalculate the mean of the six dimension scores. Apply the grade mapping:\n\n| Grade | Condition |\n|-------|-----------|\n| **Strong Accept** | mean ≥ 4.5 AND no dimension \u003C 3 |\n| **Accept** | mean ≥ 3.8 AND no dimension \u003C 2 |\n| **Weak Accept** | mean ≥ 3.0 AND no dimension \u003C 2 |\n| **Weak Reject** | mean ≥ 2.0 AND (mean \u003C 3.0 OR any dimension \u003C 2) |\n| **Reject** | mean \u003C 2.0 OR any dimension = 1 |\n\n### Step 7: Write Report\n\nWrite `level2_report.json` to the artifact root:\n\n```json\n{\n  \"artifact\": \"\u003Cname>\",\n  \"artifact_dir\": \"\u003Cpath>\",\n  \"review_version\": \"3.0.0\",\n  \"prerequisite\": \"Level 1 passed\",\n\n  \"overall\": {\n    \"grade\": \"Accept\",\n    \"mean_score\": 4.1,\n    \"one_line_summary\": \"\u003C1 sentence: what makes this ARA strong or weak>\",\n    \"strengths_summary\": [\"\u003Ctop 2-3 strengths across all dimensions>\"],\n    \"weaknesses_summary\": [\"\u003Ctop 2-3 weaknesses across all dimensions>\"]\n  },\n\n  \"dimensions\": {\n    \"D1_evidence_relevance\": {\n      \"score\": 4,\n      \"strengths\": [\"Evidence is substantively relevant for all 6 claims\"],\n      \"weaknesses\": [\"C02 cites a correlation study but makes a causal claim\"],\n      \"suggestions\": [\"Add an ablation experiment to isolate the causal mechanism for C02\"]\n    },\n    \"D2_falsifiability\": {\n      \"score\": 4,\n      \"strengths\": [\"...\"],\n      \"weaknesses\": [\"C02 falsification criteria is hard to operationalize independently\"],\n      \"suggestions\": [\"Specify a concrete re-annotation protocol for C02\"]\n    },\n    \"D3_scope_calibration\": { \"score\": 4, \"...\" : \"...\" },\n    \"D4_argument_coherence\": { \"score\": 4, \"...\" : \"...\" },\n    \"D5_exploration_integrity\": { \"score\": 3, \"...\" : \"...\" },\n    \"D6_methodological_rigor\": { \"score\": 4, \"...\" : \"...\" }\n  },\n\n  \"findings\": [\n    {\n      \"finding_id\": \"F01\",\n      \"dimension\": \"D6_methodological_rigor\",\n      \"severity\": \"major\",\n      \"target_file\": \"logic\u002Fexperiments.md\",\n      \"target_entity\": \"E03\",\n      \"evidence_span\": \"**Baselines**: No random or retrieval-only baseline reported\",\n      \"observation\": \"E03 evaluates four LLMs on research ideation but includes no non-LLM baseline.\",\n      \"reasoning\": \"Without a random or retrieval-only baseline, it is impossible to assess whether LLM performance is meaningfully above chance.\",\n      \"suggestion\": \"Add a retrieval-only baseline (e.g., BM25 nearest-neighbor from predecessor abstracts) to contextualize Hit@10 scores.\"\n    }\n  ],\n\n  \"questions_for_authors\": [\n    \"What is the inter-annotator agreement on thinking-pattern classification? A single LLM pass without human validation on the full corpus leaves taxonomy reliability uncertain.\",\n    \"...\"\n  ],\n\n  \"read_order\": [\"PAPER.md\", \"logic\u002Fclaims.md\", \"...\"]\n}\n```\n\n---\n\n## Critical Rules\n\n1. **Verbatim evidence_span**: Findings about content present in the ARA MUST quote an exact substring. Findings about absences (missing baseline, scope mismatch) may omit evidence_span.\n\n2. **Constructive tone**: Every weakness must come with a suggestion. You are helping authors improve, not punishing them.\n\n3. **Calibrated scoring**: Most competent ARAs should land in the 3-4 range. A score of 5 means genuinely excellent, not just \"no problems found.\" A score of 1 means fundamental problems, not just \"could be better.\"\n\n4. **No false grounding**: Support must flow through Proof → experiments.md → evidence\u002F. Agreement in prose (problem.md, architecture.md) does not substitute for experimental evidence.\n\n5. **Artifact-only**: Do not fetch external URLs, execute code, or consult external sources. Take the ARA's reported evidence at face value.\n\n6. **Balanced review**: Actively look for strengths, not just weaknesses. A review that only lists problems is not useful.\n\n7. **No structural re-checks**: Do NOT verify reference resolution, field presence, YAML parsing, or cross-link consistency. Level 1 has already validated all of this. Focus entirely on whether the *content* is epistemically sound.\n\n---\n\n## Reference\n\nSee [references\u002Freview-dimensions.md](references\u002Freview-dimensions.md) for scoring anchor details and check inventories per dimension.\n","# ARA Seal Level 2: Semantic Epistemic Review\n\nYou are an objective research reviewer for Agent-Native Research Artifacts. You receive an\nARA directory path and produce a comprehensive review as `level2_report.json` at the\nartifact root. You operate entirely through your native tools (Read, Write, Glob, Grep).\nYou do NOT execute code, fetch URLs, or consult external sources.\n\n**Prerequisite**: Level 1 (structural validation) has already passed. All references\nresolve, required fields exist, the exploration tree parses correctly, and cross-layer\nlinks are bidirectionally consistent. Level 2 does NOT re-check any of this. Instead, it\nevaluates whether the *content* of the ARA is epistemically sound: whether evidence\nactually supports claims, whether the argument is coherent, and whether the research\nprocess is honestly documented.\n\nYour review is **constructive**: identify both strengths and weaknesses, provide actionable\nsuggestions, and give a calibrated overall assessment. You are not a bug detector; you are\na reviewer who helps authors improve their work.\n\n---\n\n## Six Review Dimensions\n\nEach dimension is scored 1-5 and includes strengths, weaknesses, and suggestions.\nAll checks are semantic: they require reading comprehension and reasoning, not structural validation.\n\n| Dimension | What it evaluates |\n|-----------|-------------------|\n| **D1. Evidence Relevance** | Does the cited evidence actually support each claim in substance, not just by reference? |\n| **D2. Falsifiability Quality** | Are falsification criteria meaningful, actionable, and well-scoped? |\n| **D3. Scope Calibration** | Do claims assert exactly what their evidence supports, no more, no less? |\n| **D4. Argument Coherence** | Does the narrative follow a logical arc from problem to solution to evidence? |\n| **D5. Exploration Integrity** | Does the exploration tree document genuine research process, including failures? |\n| **D6. Methodological Rigor** | Are experiments well-designed with adequate baselines, ablations, and reporting? |\n\n---\n\n## Procedure\n\n### Step 1: Read the ARA\n\nRead files in this fixed order. Record the list as `read_order` in the report.\n\n1. `PAPER.md`\n2. `logic\u002Fclaims.md`\n3. `logic\u002Fexperiments.md`\n4. `logic\u002Fproblem.md`\n5. `logic\u002Fconcepts.md`\n6. `logic\u002Fsolution\u002Farchitecture.md`, `algorithm.md`, `constraints.md`, `heuristics.md`\n7. `logic\u002Frelated_work.md`\n8. `trace\u002Fexploration_tree.yaml`\n9. `evidence\u002FREADME.md` (if exists)\n10. Spot-check 2-3 evidence files from `evidence\u002Ftables\u002F` or `evidence\u002Ffigures\u002F`\n\n### Step 2: Parse Entities\n\n**Claims** (from `logic\u002Fclaims.md`): each `## C{NN}: {title}` section. Extract:\n- `Statement`, `Status`, `Falsification criteria`, `Proof` (experiment IDs), `Dependencies` (claim IDs), `Tags`\n\n**Experiments** (from `logic\u002Fexperiments.md`): each `## E{NN}: {title}` section. Extract:\n- `Verifies` (claim IDs), `Setup`, `Procedure`, `Metrics`, `Expected outcome`, `Baselines`, `Dependencies`\n\n**Heuristics** (from `logic\u002Fsolution\u002Fheuristics.md`): each `## H{NN}` section. Extract:\n- `Rationale`, `Sensitivity`, `Bounds`, `Code ref`\n\n**Observations and Gaps** (from `logic\u002Fproblem.md`): each `O{N}` and `G{N}`.\n\n**Exploration tree** (from `trace\u002Fexploration_tree.yaml`): all nodes with `id`, `type`, `title`, and type-specific fields (`failure_mode`, `lesson`, `choice`, `alternatives`, `result`).\n\n### Step 3: Build Working Maps\n\nConstruct these maps as inputs for semantic analysis. Do NOT validate structural integrity\n(Level 1 guarantees it).\n\n- **claim_proof_map**: for each claim, the set of experiment IDs in its Proof\n- **experiment_verifies_map**: for each experiment, the set of claim IDs in its Verifies\n- **claim_dependency_edges**: directed edges from each claim to its Dependencies\n- **gap_set**: all G{N} from problem.md\n- **rejected_nodes**: exploration tree nodes with type = `dead_end` or `pivot`\n- **decision_nodes**: exploration tree nodes with type = `decision`\n\n### Step 4: Evaluate Each Dimension\n\nFor each dimension, perform semantic reasoning over the parsed content. Record strengths, weaknesses, and suggestions as you go.\n\n---\n\n#### D1. Evidence Relevance\n\nFor each claim-experiment pair linked through Proof\u002FVerifies:\n\n- **Relevance**: Does the experiment's Setup\u002FProcedure\u002FMetrics actually address what the claim asserts? (Not just \"link exists\" but \"link is substantively relevant.\")\n- **Type-aware entailment**: Infer claim type from Statement cues, check experiment design matches:\n  - Causal (\"causes\", \"leads to\", \"enables\") → needs isolating ablation\n  - Generalization (\"generalizes\", \"robust\", \"across\") → needs heterogeneous test conditions\n  - Improvement (\"outperforms\", \"better\", \"improves\") → needs baseline comparison\n  - Descriptive (\"accounts for\", \"distribution\", \"pattern\") → needs representative sampling\n  - Scoping (\"when\", \"under conditions\", \"limited to\") → needs declared bounds\n- **Evidence sufficiency**: Is a single experiment enough to support this claim, or does the claim's scope demand multiple independent experiments?\n\n**Scoring anchors:**\n- **5**: Type-appropriate, relevant evidence for every claim; multi-experiment support where needed\n- **4**: Evidence relevant for all claims, minor type mismatches (e.g., causal claim with correlation-only evidence)\n- **3**: Most claim-experiment pairs are relevant, 1-2 weak matches where evidence doesn't quite address the claim\n- **2**: Multiple claims where cited experiments don't substantively address what the claim asserts\n- **1**: Majority of claims cite experiments that are irrelevant to their statements\n\n---\n\n#### D2. Falsifiability Quality\n\nFor each claim's Falsification criteria field:\n\n- **Actionability**: Could an independent researcher execute this criterion? Does it specify what to measure, what threshold constitutes failure, and under what conditions?\n- **Non-triviality**: Is the criterion non-tautological? (\"If the method doesn't work\" is trivial. \"Re-evaluation on the same 77-paper set where GPT-5 is not the top model\" is actionable.)\n- **Scope match**: Does the falsification criterion address the same scope as the Statement? (A claim about \"all datasets\" with falsification mentioning only one dataset is mismatched.)\n- **Independence**: Could the criterion be tested without access to the authors' proprietary data or systems?\n\n**Scoring anchors:**\n- **5**: Every claim has specific, actionable, independently testable falsification criteria matching the claim's scope\n- **4**: Most criteria are strong, 1-2 are vague or hard to operationalize\n- **3**: Mixed quality; some actionable, some trivial or scope-mismatched\n- **2**: Most criteria are trivial, tautological, or scope-mismatched\n- **1**: Falsification criteria meaningless across claims\n\n---\n\n#### D3. Scope Calibration\n\n- **Over-claiming**: Does any Statement use universal scope markers (\"all models\", \"any dataset\", \"state-of-the-art across all\") while cited experiments cover only specific, narrow conditions? The gap must be substantial.\n- **Under-claiming**: Are there important experimental results present in evidence\u002F that are not captured by any claim? (Evidence without a corresponding claim.)\n- **Assumption explicitness**: Are key assumptions stated in problem.md (Assumptions section) or constraints.md? Are there unstated assumptions implied by the experimental design?\n- **Generalization boundaries**: Does the artifact clearly state what the claims do NOT apply to? Check constraints.md and limitations in the exploration tree.\n- **Qualifier consistency**: When claims use hedging (\"tends to\", \"in most cases\"), is this consistent with the evidence strength?\n\n**Scoring anchors:**\n- **5**: All claims precisely match evidence scope, assumptions explicit, limits clearly stated\n- **4**: Claims well-scoped with minor gaps in assumption documentation\n- **3**: Some claims slightly over\u002Funder-reach, assumptions partially stated\n- **2**: Multiple over-claims or significant undocumented assumptions\n- **1**: Pervasive scope mismatch between claims and evidence\n\n---\n\n#### D4. Argument Coherence\n\n- **Observation → Gap derivation**: Do the stated gaps follow logically from the observations? Or are they asserted without connection?\n- **Gap → Insight connection**: Does the key insight in problem.md address the identified gaps?\n- **Insight → Solution alignment**: Does the solution architecture implement the key insight?\n- **Solution → Claims coverage**: Do the claims cover the solution's main contributions?\n- **Cross-layer consistency**: Do claims, exploration tree, and evidence tell the same story? Flag contradictions.\n- **Narrative completeness**: Are there motivating questions from problem.md that are neither answered nor explicitly deferred?\n- **Gap coverage**: For each gap in problem.md, is there at least one claim that substantively addresses it? Flag gaps that are motivated but never resolved.\n\n**Scoring anchors:**\n- **5**: Clear logical arc (observations → gaps → insight → solution → claims → evidence), all gaps addressed, no contradictions\n- **4**: Strong flow with minor logical gaps or one unaddressed gap\n- **3**: General flow present but some disconnects between layers\n- **2**: Significant misalignment between problem statement and claims, or unresolved contradictions\n- **1**: No coherent logical flow; layers tell different stories\n\n---\n\n#### D5. Exploration Integrity\n\n- **Dead-end quality**: Is the `failure_mode` specific enough to be actionable? (\"Didn't work\" is bad. \"Divergence after 1000 steps due to gradient explosion\" is good.) Is the `lesson` a genuine transferable insight?\n- **Decision rationale quality**: Do rationales explain WHY the chosen path was preferred over alternatives? Are alternatives real alternatives or strawmen?\n- **Rebutted-branch consistency**: Does any claim advocate an approach marked as dead_end or pivot in the tree? (This is a logical contradiction.)\n- **Exploration breadth**: For the paper's main design choices, were at least 2 alternatives considered and documented?\n- **Honesty signal**: Does the tree document genuine negative results, or does it read like a post-hoc justification? A tree with zero dead-ends or only trivial failures is suspicious.\n\n**Scoring anchors:**\n- **5**: Rich tree with well-documented dead-ends (specific failure modes, actionable lessons), thorough decision rationale, genuine negative results\n- **4**: Good tree with minor gaps in dead-end documentation or decision rationale\n- **3**: Tree present but dead-ends lack specificity or decisions lack alternatives\n- **2**: Boilerplate documentation; dead-ends and decisions read as formulaic rather than authentic\n- **1**: Tree contradicts claims or reads entirely as post-hoc justification\n\n---\n\n#### D6. Methodological Rigor\n\n- **Baseline adequacy**: Are the right things being compared? Are baselines recent and relevant? Flag experiments with \"no baseline\" for comparative claims.\n- **Ablation coverage**: For claims involving multiple components, does at least one experiment isolate individual contributions?\n- **Statistical reporting**: Do experiments mention variance, confidence intervals, number of runs, or statistical tests? Flag single-run results for quantitative claims.\n- **Metric-claim alignment**: Does the metric actually measure what the claim asserts? (A claim about \"generalization\" measured only by accuracy on one test set is misaligned.)\n- **Reproducibility signals**: Are experiment setups specific enough for independent replication? (Model name, dataset, hardware, hyperparameters.)\n\n**Scoring anchors:**\n- **5**: Comprehensive baselines, proper ablations, statistical rigor, metrics precisely match claims, fully reproducible setup\n- **4**: Strong methodology with minor gaps (e.g., missing variance on one experiment)\n- **3**: Adequate but missing some baselines or statistical details\n- **2**: Significant gaps; missing baselines for comparative claims or no ablations\n- **1**: No baselines, no ablations, metrics don't match claims\n\n---\n\n### Step 5: Compile Findings\n\nCollect all issues found across the six dimensions into a single findings list. Assign each finding:\n\n- **finding_id**: F01, F02, ... (sequential)\n- **dimension**: which of D1-D6\n- **severity**: one of:\n  - `critical` — fundamental epistemic flaw; the claim or argument cannot stand as written\n  - `major` — significant weakness that undermines a claim or dimension score\n  - `minor` — noticeable issue that doesn't invalidate the work\n  - `suggestion` — constructive improvement opportunity, not a flaw\n- **target_file**: which ARA file\n- **target_entity**: C{NN}, E{NN}, H{NN}, G{N}, or node ID (if applicable)\n- **evidence_span**: verbatim substring from the ARA that triggered the finding (MUST be exact quote; omit if the finding is about an absence)\n- **observation**: what you found (factual)\n- **reasoning**: why it matters (analytical)\n- **suggestion**: how to fix or improve it (constructive)\n\nSort findings by severity: critical first, then major, minor, suggestion.\n\n### Step 6: Compute Overall Grade\n\nCalculate the mean of the six dimension scores. Apply the grade mapping:\n\n| Grade | Condition |\n|-------|-----------|\n| **Strong Accept** | mean ≥ 4.5 AND no dimension \u003C 3 |\n| **Accept** | mean ≥ 3.8 AND no dimension \u003C 2 |\n| **Weak Accept** | mean ≥ 3.0 AND no dimension \u003C 2 |\n| **Weak Reject** | mean ≥ 2.0 AND (mean \u003C 3.0 OR any dimension \u003C 2) |\n| **Reject** | mean \u003C 2.0 OR any dimension = 1 |\n\n### Step 7: Write Report\n\nWrite `level2_report.json` to the artifact root:\n\n```json\n{\n  \"artifact\": \"\u003Cname>\",\n  \"artifact_dir\": \"\u003Cpath>\",\n  \"review_version\": \"3.0.0\",\n  \"prerequisite\": \"Level 1 passed\",\n\n  \"overall\": {\n    \"grade\": \"Accept\",\n    \"mean_score\": 4.1,\n    \"one_line_summary\": \"\u003C1 sentence: what makes this ARA strong or weak>\",\n    \"strengths_summary\": [\"\u003Ctop 2-3 strengths across all dimensions>\"],\n    \"weaknesses_summary\": [\"\u003Ctop 2-3 weaknesses across all dimensions>\"]\n  },\n\n  \"dimensions\": {\n    \"D1_evidence_relevance\": {\n      \"score\": 4,\n      \"strengths\": [\"Evidence is substantively relevant for all 6 claims\"],\n      \"weaknesses\": [\"C02 cites a correlation study but makes a causal claim\"],\n      \"suggestions\": [\"Add an ablation experiment to isolate the causal mechanism for C02\"]\n    },\n    \"D2_falsifiability\": {\n      \"score\": 4,\n      \"strengths\": [\"...\"],\n      \"weaknesses\": [\"C02 falsification criteria is hard to operationalize independently\"],\n      \"suggestions\": [\"Specify a concrete re-annotation protocol for C02\"]\n    },\n    \"D3_scope_calibration\": { \"score\": 4, \"...\" : \"...\" },\n    \"D4_argument_coherence\": { \"score\": 4, \"...\" : \"...\" },\n    \"D5_exploration_integrity\": { \"score\": 3, \"...\" : \"...\" },\n    \"D6_methodological_rigor\": { \"score\": 4, \"...\" : \"...\" }\n  },\n\n  \"findings\": [\n    {\n      \"finding_id\": \"F01\",\n      \"dimension\": \"D6_methodological_rigor\",\n      \"severity\": \"major\",\n      \"target_file\": \"logic\u002Fexperiments.md\",\n      \"target_entity\": \"E03\",\n      \"evidence_span\": \"**Baselines**: No random or retrieval-only baseline reported\",\n      \"observation\": \"E03 evaluates four LLMs on research ideation but includes no non-LLM baseline.\",\n      \"reasoning\": \"Without a random or retrieval-only baseline, it is impossible to assess whether LLM performance is meaningfully above chance.\",\n      \"suggestion\": \"Add a retrieval-only baseline (e.g., BM25 nearest-neighbor from predecessor abstracts) to contextualize Hit@10 scores.\"\n    }\n  ],\n\n  \"questions_for_authors\": [\n    \"What is the inter-annotator agreement on thinking-pattern classification? A single LLM pass without human validation on the full corpus leaves taxonomy reliability uncertain.\",\n    \"...\"\n  ],\n\n  \"read_order\": [\"PAPER.md\", \"logic\u002Fclaims.md\", \"...\"]\n}\n```\n\n---\n\n## Critical Rules\n\n1. **Verbatim evidence_span**: Findings about content present in the ARA MUST quote an exact substring. Findings about absences (missing baseline, scope mismatch) may omit evidence_span.\n\n2. **Constructive tone**: Every weakness must come with a suggestion. You are helping authors improve, not punishing them.\n\n3. **Calibrated scoring**: Most competent ARAs should land in the 3-4 range. A score of 5 means genuinely excellent, not just \"no problems found.\" A score of 1 means fundamental problems, not just \"could be better.\"\n\n4. **No false grounding**: Support must flow through Proof → experiments.md → evidence\u002F. Agreement in prose (problem.md, architecture.md) does not substitute for experimental evidence.\n\n5. **Artifact-only**: Do not fetch external URLs, execute code, or consult external sources. Take the ARA's reported evidence at face value.\n\n6. **Balanced review**: Actively look for strengths, not just weaknesses. A review that only lists problems is not useful.\n\n7. **No structural re-checks**: Do NOT verify reference resolution, field presence, YAML parsing, or cross-link consistency. Level 1 has already validated all of this. Focus entirely on whether the *content* is epistemically sound.\n\n---\n\n## Reference\n\nSee [references\u002Freview-dimensions.md](references\u002Freview-dimensions.md) for scoring anchor details and check inventories per dimension.\n",{"name":5,"tags":24,"author":33,"license":15,"version":34,"description":7,"dependencies":35},[25,26,27,28,29,30,31,32],"ARA","Epistemic Review","Research Rigor","Peer Review","Scoring","Audit","Falsifiability","Research Tooling","Orchestra Research","3.0.0",[],"skills","4fb202b97f2676132ac8713dbd2370cefacacb81",".agents\u002Fskills\u002Frigor-reviewer","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills","https:\u002F\u002Fgithub.com\u002Fzengrong233","https:\u002F\u002Fwtu.edu.cn\u002F","my name songzengrong\r\n  research interest's:NLP","2026-06-04 10:48:22",{"source":8,"skillName":5,"commands":45,"prompt":55,"zipAvailable":56},[46,49,52],{"tool":47,"command":48},"npx","npx skills add zengrong233\u002Fdevin-skills --skill ara-rigor-reviewer",{"tool":50,"command":51},"bunx","bunx skills add zengrong233\u002Fdevin-skills --skill ara-rigor-reviewer",{"tool":53,"command":54},"pnpm","pnpm dlx skills add zengrong233\u002Fdevin-skills --skill ara-rigor-reviewer","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Frigor-reviewer\u002FSKILL.md",true,[58,64],{"path":59,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":62,"sourceHtmlUrl":63},".openskills.json","REFERENCE","参考文档",254,"https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Frigor-reviewer\u002F.openskills.json",{"path":65,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Freview-dimensions.md",8309,"https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Frigor-reviewer\u002Freferences\u002Freview-dimensions.md",[69,76,83,92,100,107],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M759188DV0S9ARY1ETHCEY45","whisper","OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fwhisper\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M75913ZB4EDVWYGYRTRWTKD1","weights-and-biases","Track ML experiments with automatic logging, visualize training in real-time, optimize hyperparameters with sweeps, and manage model registry with W&B - collaborative MLOps platform","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":87,"licenseFileUrl":88,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":19},"01M75910YNW7H0KZQDR96TGWZ9","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":87,"licenseFileUrl":96,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M7590X6XJR78KVSHCVFYB59A","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M7590RENMJJTXV44P8G553ZQ","serving-llms-vllm","Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency\u002Fthroughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ\u002FAWQ\u002FFP8), and tensor parallelism.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fvllm\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M7590GA6EZKJA8GAAMMPAXBB","verl-rl-training","Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fverl\u002FSKILL.md",[],[]]