[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6HPRGSBFP1N6SFQFC52SYFR":3,"skill-same:01M6HPRGSBFP1N6SFQFC52SYFR":97},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":26,"publishedAt":49,"linkOnly":50,"markdownContent":51,"markdownBody":52,"frontMatter":53,"refName":74,"commitSha":75,"skillDirectory":76,"forksCount":77,"repositoryHtmlUrl":78,"repositoryHomepageUrl":6,"ownerHtmlUrl":79,"ownerBlogUrl":80,"ownerDescription":81,"updatedAt":82,"install":83,"resources":96},"01M6HPRGSBFP1N6SFQFC52SYFR","gfxstream_deqp_vk",null,"Build, run, debug, and iterate on Linux deqp-vk Vulkan CTS tests over Gfxstream (guest vulkan.ranchu.so + host kumquat_virtio + Lavapipe) without an Android guest image. Use this skill whenever running dEQP-VK tests against Gfxstream, diagnosing Vulkan CTS failures, collecting guest\u002Fhost\u002FQPA logs, or making iterative fixes to Gfxstream guest (external\u002Fmesa3d), Gfxstream host (hardware\u002Fgoogle\u002Fgfxstream), or dEQP (external\u002Fdeqp). This skill assumes a certain checkout structure to locate gfxstream, mesa3d, and deqp folders relative to the workspace root.","google\u002Fgfxstream","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M6HPRE9KHA311A3B04BX5XFY",65,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fgfxstream\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fgfxstream_deqp_vk\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,9],"gfxstream deqp vk","vk","structure","running","Workspace Root","iterate","fixes","root","locate","checkout","host","tests","image","build","use","workspace","logs","run","debug","linux","vulkan","android","2026-11-10 10:12:30",false,"---\nname: ml-evaluation-and-tracking\ndescription: \"Benchmark and track machine learning experiments: lm-eval-harness for LLM benchmarks and Weights & Biases for experiment logging.\"\nversion: 1.0.0\nauthor: Hermes Agent\nlicense: MIT\nplatforms: [linux, macos, windows]\nmetadata:\n  hermes:\n    tags: [ml, evaluation, benchmarking, wandb, experiment-tracking, llm]\n    related_skills: [llm-inference-and-serving, huggingface-hub]\n---\n\n# ML Evaluation and Tracking\n\n## Evaluating LLMs (lm-eval-harness)\n\nRun standardized benchmarks (MMLU, GSM8K, HellaSwag, etc.) against any HuggingFace-compatible model or local API endpoint.\n\n### Install\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\ncd lm-evaluation-harness\npip install -e .\n```\n\n### Run a Benchmark\n```bash\nlm_eval --model hf --model_args pretrained=meta-llama\u002FLlama-2-7b-hf --tasks mmlu,gsm8k --batch_size 8 --device cuda\n```\n\n### Tasks Available\n| Task | Measures |\n|------|----------|\n| `mmlu` | Multi-subject knowledge |\n| `gsm8k` | Grade-school math reasoning |\n| `hellaswag` | Commonsense inference |\n| `arc_challenge` | Science questions |\n| `truthfulqa_mc` | Truthfulness |\n\n### Output Formats\n```bash\nlm_eval ... --output_path .\u002Fresults --log_samples\n# Produces JSON with aggregated scores and per-sample details\n```\n\n### Custom Tasks\nAdd YAML task definitions to `lm_eval\u002Ftasks\u002F`:\n```yaml\ndataset_path: json\nvalidation_split: validation\nprocess_docs: !function utils.process\ndoc_to_text: \"{{question}}\\nA. {{choices[0]}}\\nB. {{choices[1]}}\"\ndoc_to_target: \"{{answer}}\"\nmetric:\n  - accuracy\n```\n\n### API Models\n```bash\nlm_eval --model local-completions --model_args base_url=http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fcompletions,num_concurrent=8 --tasks mmlu\n```\n\n## Weights & Biases (W&B)\n\nLog experiments, track metrics, manage model registry, and build dashboards.\n\n### Setup\n```bash\npip install wandb\nwandb login\n```\n\n### Basic Logging\n```python\nimport wandb\nwandb.init(project=\"my-project\", config={\"lr\": 0.001, \"epochs\": 10})\nfor epoch in range(10):\n    loss = train_step()\n    wandb.log({\"loss\": loss, \"epoch\": epoch})\nwandb.finish()\n```\n\n### Artifacts (Model Versioning)\n```python\nartifact = wandb.Artifact(\"model\", type=\"model\")\nartifact.add_dir(\".\u002Fcheckpoints\")\nwandb.log_artifact(artifact)\n```\n\n### Sweeps (Hyperparameter Search)\n```yaml\n# sweep.yaml\nprogram: train.py\nmethod: bayes\nmetric:\n  name: val_loss\n  goal: minimize\nparameters:\n  lr:\n    distribution: uniform\n    min: 0.0001\n    max: 0.01\n```\n```bash\nwandb sweep sweep.yaml\nwandb agent sweep-id\n```\n\n### Common Pitfalls\n- `lm-eval-harness` task names are case-sensitive\n- W&B offline mode requires `wandb sync` later; don't forget\n- Large batch sizes in lm-eval can OOM on consumer GPUs — start with 1 and scale up\n- Always pin `lm-eval-harness` to a specific git commit for reproducibility\n","# ML Evaluation and Tracking\n\n## Evaluating LLMs (lm-eval-harness)\n\nRun standardized benchmarks (MMLU, GSM8K, HellaSwag, etc.) against any HuggingFace-compatible model or local API endpoint.\n\n### Install\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness\ncd lm-evaluation-harness\npip install -e .\n```\n\n### Run a Benchmark\n```bash\nlm_eval --model hf --model_args pretrained=meta-llama\u002FLlama-2-7b-hf --tasks mmlu,gsm8k --batch_size 8 --device cuda\n```\n\n### Tasks Available\n| Task | Measures |\n|------|----------|\n| `mmlu` | Multi-subject knowledge |\n| `gsm8k` | Grade-school math reasoning |\n| `hellaswag` | Commonsense inference |\n| `arc_challenge` | Science questions |\n| `truthfulqa_mc` | Truthfulness |\n\n### Output Formats\n```bash\nlm_eval ... --output_path .\u002Fresults --log_samples\n# Produces JSON with aggregated scores and per-sample details\n```\n\n### Custom Tasks\nAdd YAML task definitions to `lm_eval\u002Ftasks\u002F`:\n```yaml\ndataset_path: json\nvalidation_split: validation\nprocess_docs: !function utils.process\ndoc_to_text: \"{{question}}\\nA. {{choices[0]}}\\nB. {{choices[1]}}\"\ndoc_to_target: \"{{answer}}\"\nmetric:\n  - accuracy\n```\n\n### API Models\n```bash\nlm_eval --model local-completions --model_args base_url=http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fcompletions,num_concurrent=8 --tasks mmlu\n```\n\n## Weights & Biases (W&B)\n\nLog experiments, track metrics, manage model registry, and build dashboards.\n\n### Setup\n```bash\npip install wandb\nwandb login\n```\n\n### Basic Logging\n```python\nimport wandb\nwandb.init(project=\"my-project\", config={\"lr\": 0.001, \"epochs\": 10})\nfor epoch in range(10):\n    loss = train_step()\n    wandb.log({\"loss\": loss, \"epoch\": epoch})\nwandb.finish()\n```\n\n### Artifacts (Model Versioning)\n```python\nartifact = wandb.Artifact(\"model\", type=\"model\")\nartifact.add_dir(\".\u002Fcheckpoints\")\nwandb.log_artifact(artifact)\n```\n\n### Sweeps (Hyperparameter Search)\n```yaml\n# sweep.yaml\nprogram: train.py\nmethod: bayes\nmetric:\n  name: val_loss\n  goal: minimize\nparameters:\n  lr:\n    distribution: uniform\n    min: 0.0001\n    max: 0.01\n```\n```bash\nwandb sweep sweep.yaml\nwandb agent sweep-id\n```\n\n### Common Pitfalls\n- `lm-eval-harness` task names are case-sensitive\n- W&B offline mode requires `wandb sync` later; don't forget\n- Large batch sizes in lm-eval can OOM on consumer GPUs — start with 1 and scale up\n- Always pin `lm-eval-harness` to a specific git commit for reproducibility\n",{"name":54,"author":55,"license":56,"version":57,"metadata":58,"platforms":70,"description":73},"ml-evaluation-and-tracking","Hermes Agent","MIT","1.0.0",{"hermes":59},{"tags":60,"related_skills":67},[61,62,63,64,65,66],"ml","evaluation","benchmarking","wandb","experiment-tracking","llm",[68,69],"llm-inference-and-serving","huggingface-hub",[46,71,72],"macos","windows","Benchmark and track machine learning experiments: lm-eval-harness for LLM benchmarks and Weights & Biases for experiment logging.","main","64e2fc9afa7bb3da1138eb73399144d0e7e73e9d",".agents\u002Fskills\u002Fgfxstream_deqp_vk",50,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fgfxstream","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-09 20:17:46",{"source":8,"skillName":54,"commands":84,"prompt":94,"zipAvailable":95},[85,88,91],{"tool":86,"command":87},"npx","npx skills add google\u002Fgfxstream --skill ml-evaluation-and-tracking",{"tool":89,"command":90},"bunx","bunx skills add google\u002Fgfxstream --skill ml-evaluation-and-tracking",{"tool":92,"command":93},"pnpm","pnpm dlx skills add google\u002Fgfxstream --skill ml-evaluation-and-tracking","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fgfxstream\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fgfxstream_deqp_vk\u002FSKILL.md",true,[],[]]