[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EKXMQDZJ2GX5EDEX4T0DPT":3,"skill-same:01M6EKXMQDZJ2GX5EDEX4T0DPT":77},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":35,"linkOnly":36,"markdownContent":37,"markdownBody":38,"frontMatter":39,"refName":54,"commitSha":55,"skillDirectory":56,"forksCount":57,"repositoryHtmlUrl":58,"repositoryHomepageUrl":6,"ownerHtmlUrl":59,"ownerBlogUrl":60,"ownerDescription":61,"updatedAt":62,"install":63,"resources":76},"01M6EKXMQDZJ2GX5EDEX4T0DPT","inner-cv-tuning-discipline",null,"Hyperparameter tuning uses 5-fold inner CV on the training set only; group-aware folds when cluster IDs are available.","cgpan\u002Fedm-ars-public","cgpan","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F118746117?v=4","01M6EKT5TG36Q3GH340QN8RH08","Chenguang Pan","01M6EKT5YTQVRARGJF0VPA1SY7",5,"MIT","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Fskills\u002Fmethodology\u002Finner-cv-tuning-discipline\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34],"tuning","discipline","cv","inner cv tuning discipline","training","validation","methodology","SET","ids","critic","only","Fold","analyst","cross-validation","cluster","Hyperparameter Tuning","2026-11-10 10:12:30",false,"---\nname: inner-cv-tuning-discipline\nlayer: methodology\ndescription: Hyperparameter tuning uses 5-fold inner CV on the training set only; group-aware folds when cluster IDs are available.\ntrigger_keywords:\n  - tuning\n  - hyperparameter\n  - cv\n  - cross-validation\n  - gridsearch\n  - groupkfold\napplicable_task_types: []\napplicable_datasets: []\napplicable_stages:\n  - Analyst\n  - Critic\npriority: 1\nreferences_skills: []\nresources: []\nversion: \"1.0\"\nrule_severity: mandatory\n---\n\n# Inner-CV Tuning Discipline\n\nHyperparameters are tuned via **5-fold inner cross-validation on the\ntraining set only**. The test set is never touched during tuning — it\nexists solely for final evaluation reporting.\n\nWhen cluster IDs (e.g., `train_school_ids.csv`) are available, the inner\nCV must be group-aware to prevent within-cluster leakage during tuning.\n\n## Group-aware inner CV\n\n```python\nfrom sklearn.model_selection import GroupKFold, StratifiedGroupKFold\nfrom sklearn.model_selection import GridSearchCV\nimport pandas as pd\n\ntrain_school_ids = pd.read_csv(\"train_school_ids.csv\")[\"pseudo_school_id\"].values\n\nif is_classification:\n    inner_cv = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)\n    cv_splits = list(inner_cv.split(train_X, train_y, train_school_ids))\nelse:\n    inner_cv = GroupKFold(n_splits=5)\n    cv_splits = list(inner_cv.split(train_X, train_y, train_school_ids))\n\ngrid_search = GridSearchCV(\n    estimator,\n    param_grid,\n    cv=cv_splits,\n    scoring=scoring,\n)\ngrid_search.fit(train_X, train_y)\n```\n\nIf `train_school_ids.csv` does not exist, fall back to plain `cv=5`. The\nAnalyst should log a warning when this fallback is taken so reviewers\nknow clustering was not respected during tuning.\n\n## Hard rules\n\n1. **Never use the test set during tuning**, including for early-stopping\n   validation. Use a portion of the training set (e.g.,\n   `validation_fraction=0.1` inside MLP's built-in early stopping) when\n   needed.\n2. **Random seed `random_state=42`** for every stochastic operation in\n   tuning (CV shuffle, classifier initialization, GridSearchCV's internal\n   randomization where applicable).\n3. **Refit on the full training set** with the best hyperparameters before\n   computing test-set metrics.\n4. **No tuning grids on the test partition.** The Critic will flag any\n   evidence of test-set use during tuning as a critical issue.\n\n## Verification rules (Critic)\n\nThe Critic verifies via `results.json` metadata or warnings that:\n\n1. Tuning used training data only. If `results.warnings` or\n   `results.errors` mention test-set leakage during tuning → critical.\n2. Tuning information is present for the tunable models (RF, XGBoost,\n   ElasticNet, MLP). If absent → major (tuning should be documented in\n   results metadata).\n3. `random_state=42` was used (minor if missing).\n\n## Source provenance\n\nCanonical source: `agent_prompts\u002Fanalyst.yaml` §\"Hyperparameter Tuning\"\n(complete code path with group-aware CV and the test-set-leakage\nprohibition).\n\nMerged content from:\n- `data_registry\u002Ftask_templates\u002Fprediction.yaml` substeps `mt_02`\n  through `mt_06` and check `check_mt_train_only` (workflow + verification\n  language)\n- `data_registry\u002Fevaluation_rubrics\u002Fmethodological_checklist.yaml` item\n  `an_02` (Critic verification)\n","# Inner-CV Tuning Discipline\n\nHyperparameters are tuned via **5-fold inner cross-validation on the\ntraining set only**. The test set is never touched during tuning — it\nexists solely for final evaluation reporting.\n\nWhen cluster IDs (e.g., `train_school_ids.csv`) are available, the inner\nCV must be group-aware to prevent within-cluster leakage during tuning.\n\n## Group-aware inner CV\n\n```python\nfrom sklearn.model_selection import GroupKFold, StratifiedGroupKFold\nfrom sklearn.model_selection import GridSearchCV\nimport pandas as pd\n\ntrain_school_ids = pd.read_csv(\"train_school_ids.csv\")[\"pseudo_school_id\"].values\n\nif is_classification:\n    inner_cv = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)\n    cv_splits = list(inner_cv.split(train_X, train_y, train_school_ids))\nelse:\n    inner_cv = GroupKFold(n_splits=5)\n    cv_splits = list(inner_cv.split(train_X, train_y, train_school_ids))\n\ngrid_search = GridSearchCV(\n    estimator,\n    param_grid,\n    cv=cv_splits,\n    scoring=scoring,\n)\ngrid_search.fit(train_X, train_y)\n```\n\nIf `train_school_ids.csv` does not exist, fall back to plain `cv=5`. The\nAnalyst should log a warning when this fallback is taken so reviewers\nknow clustering was not respected during tuning.\n\n## Hard rules\n\n1. **Never use the test set during tuning**, including for early-stopping\n   validation. Use a portion of the training set (e.g.,\n   `validation_fraction=0.1` inside MLP's built-in early stopping) when\n   needed.\n2. **Random seed `random_state=42`** for every stochastic operation in\n   tuning (CV shuffle, classifier initialization, GridSearchCV's internal\n   randomization where applicable).\n3. **Refit on the full training set** with the best hyperparameters before\n   computing test-set metrics.\n4. **No tuning grids on the test partition.** The Critic will flag any\n   evidence of test-set use during tuning as a critical issue.\n\n## Verification rules (Critic)\n\nThe Critic verifies via `results.json` metadata or warnings that:\n\n1. Tuning used training data only. If `results.warnings` or\n   `results.errors` mention test-set leakage during tuning → critical.\n2. Tuning information is present for the tunable models (RF, XGBoost,\n   ElasticNet, MLP). If absent → major (tuning should be documented in\n   results metadata).\n3. `random_state=42` was used (minor if missing).\n\n## Source provenance\n\nCanonical source: `agent_prompts\u002Fanalyst.yaml` §\"Hyperparameter Tuning\"\n(complete code path with group-aware CV and the test-set-leakage\nprohibition).\n\nMerged content from:\n- `data_registry\u002Ftask_templates\u002Fprediction.yaml` substeps `mt_02`\n  through `mt_06` and check `check_mt_train_only` (workflow + verification\n  language)\n- `data_registry\u002Fevaluation_rubrics\u002Fmethodological_checklist.yaml` item\n  `an_02` (Critic verification)\n",{"name":5,"layer":25,"version":40,"priority":41,"resources":42,"description":7,"rule_severity":43,"trigger_keywords":44,"applicable_stages":48,"references_skills":51,"applicable_datasets":52,"applicable_task_types":53},"1.0",1,[],"mandatory",[19,45,21,32,46,47],"hyperparameter","gridsearch","groupkfold",[49,50],"Analyst","Critic",[],[],[],"main","1aeaead66d2bf876109f3b101005d2eed54041d9","skills\u002Fmethodology\u002Finner-cv-tuning-discipline",0,"https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public","https:\u002F\u002Fgithub.com\u002Fcgpan","https:\u002F\u002Fcpan.ai","Ph.D. candidate in Measurement, Evaluation, and Statistics.","2026-09-28 20:39:06",{"source":8,"skillName":5,"commands":64,"prompt":74,"zipAvailable":75},[65,68,71],{"tool":66,"command":67},"npx","npx skills add cgpan\u002Fedm-ars-public --skill inner-cv-tuning-discipline",{"tool":69,"command":70},"bunx","bunx skills add cgpan\u002Fedm-ars-public --skill inner-cv-tuning-discipline",{"tool":72,"command":73},"pnpm","pnpm dlx skills add cgpan\u002Fedm-ars-public --skill inner-cv-tuning-discipline","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Fskills\u002Fmethodology\u002Finner-cv-tuning-discipline\u002FSKILL.md",true,[],[78,101,114,123,135,144],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":94,"publishedAt":35},"01M6EM1ZTQZVAP90DZTVBHCCFB","acm-fixture","Fixture writing skill for ACM acmart sigconf paper output.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Fwriting\u002Facm-fixture\u002FSKILL.md",[84,89],{"categoryNo":85,"categoryName":86,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":90,"categoryName":91,"taxonomyType":92,"taxonomyTypeDesc":93},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[95,96,97,98,99,100],"acm fixture","fixture","writing","LaTeX","writer","paper",{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":35},"01M6EM1XNFETAZFGPHQPAXKBYN","prediction-fixture","Fixture skill for the prediction task type; references shap-fixture.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Ftask-type\u002Fprediction-fixture\u002FSKILL.md",[],[96,108,109,110,111,112,113,31],"prediction fixture","task","type","references","fixture skill","shap",{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":35},"01M6EM1V4MQXYGQ7K2ZDJRAPSM","causal-fixture","Fixture skill for the causal_inference task type.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Ftask-type\u002Fcausal-fixture\u002FSKILL.md",[],[96,121,110,109,122,112],"causal fixture","inference",{"skillNo":124,"name":125,"nameZh":6,"description":126,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":127,"categories":128,"keywords":129,"publishedAt":35},"01M6EM1SCWFZZ2ACSAP65GNWK9","shap-fixture","Fixture methodology for SHAP feature importance analysis.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Fmethodology\u002Fshap-fixture\u002FSKILL.md",[],[130,96,113,131,25,132,133,134],"shap fixture","feature importance","analysis","explainer","feature",{"skillNo":136,"name":137,"nameZh":6,"description":138,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":139,"categories":140,"keywords":141,"publishedAt":35},"01M6EM1PZC1MK1G0D1YFVYAQ9W","cluster-recon-fixture","Fixture methodology for reconstructing cluster IDs from fingerprint variables.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Fmethodology\u002Fcluster-recon-fixture\u002FSKILL.md",[],[142,143,33,96,27,25],"recon","cluster recon fixture",{"skillNo":145,"name":146,"nameZh":6,"description":147,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":148,"categories":149,"keywords":150,"publishedAt":35},"01M6EM1MZVKMSWEXZ7TJ45DS04","hsls09-fixture","Fixture skill for HSLS:09 with a bundled YAML resource and one cross-reference.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Fdataset\u002Fhsls09-fixture\u002FSKILL.md",[],[96,151,152,153,154,112,155,142,33,156],"hsls09 fixture","yaml","public","reference","dataset","registry"]