[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M796VBJSCVGX5D375FBPAF50":3,"skill-same:01M796VBJSCVGX5D375FBPAF50":56},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":33,"commitSha":34,"skillDirectory":35,"forksCount":36,"repositoryHtmlUrl":37,"repositoryHomepageUrl":6,"ownerHtmlUrl":38,"ownerBlogUrl":39,"ownerDescription":40,"updatedAt":41,"install":42,"resources":55},"01M796VBJSCVGX5D375FBPAF50","vastai-incident-runbook",null,"Execute Vast.ai incident response for GPU instance failures and outages.\nUse when responding to instance failures, investigating training crashes,\nor handling spot preemption emergencies.\nTrigger with phrases like \"vastai incident\", \"vastai outage\",\n\"vastai down\", \"vastai emergency\", \"vastai instance failed\".","jukeyman\u002Fjukeyman-skills","jukeyman","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F142960420?v=4","01M78FR4Y3DHMMCBCC0GXXNB3F","RICK JEFFERSON","01M78FR4Z8RVQRAD31Y5J7V6EB",1,"MIT","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fsaas-packs--vastai-pack--vastai-incident-runbook\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: vastai-incident-runbook\ndescription: |\n  Execute Vast.ai incident response for GPU instance failures and outages.\n  Use when responding to instance failures, investigating training crashes,\n  or handling spot preemption emergencies.\n  Trigger with phrases like \"vastai incident\", \"vastai outage\",\n  \"vastai down\", \"vastai emergency\", \"vastai instance failed\".\nallowed-tools: Read, Grep, Bash(vastai:*), Bash(curl:*), Bash(ssh:*)\nversion: 1.0.0\nlicense: MIT\nauthor: Jeremy Longshore \u003Cjeremy@intentsolutions.io>\ncompatible-with: claude-code, codex, openclaw\ntags: [saas, vast-ai, incident-response]\n---\n# Vast.ai Incident Runbook\n\n## Overview\nRapid incident response procedures for Vast.ai GPU instance failures. Covers triage, mitigation, recovery, and postmortem for common incident types: spot preemption, instance crashes, GPU failures, and billing issues.\n\n## Prerequisites\n- Vast.ai CLI access\n- SSH access to instances (if still running)\n- Checkpoint storage accessible (S3\u002FGCS)\n\n## Instructions\n\n### Triage: Assess Impact (\u003C 2 minutes)\n\n```bash\n#!\u002Fbin\u002Fbash\nset -euo pipefail\necho \"=== INCIDENT TRIAGE ===\"\necho \"Time: $(date -u)\"\n\n# 1. Check all instances\necho -e \"\\n--- Instance Status ---\"\nvastai show instances --raw | python3 -c \"\nimport sys, json\nfor inst in json.load(sys.stdin):\n    status = inst.get('actual_status', '?')\n    flag = 'ALERT' if status in ('error', 'exited', 'offline') else 'OK'\n    print(f'  [{flag}] ID:{inst[\\\"id\\\"]} Status:{status} '\n          f'GPU:{inst.get(\\\"gpu_name\\\",\\\"?\\\")} \\${inst.get(\\\"dph_total\\\",0):.3f}\u002Fhr')\n\"\n\n# 2. Check if affected instance has recent logs\necho -e \"\\n--- Recent Logs (last 20 lines) ---\"\nvastai logs ${INSTANCE_ID:-0} --tail 20 2>\u002Fdev\u002Fnull || echo \"No logs available\"\n\n# 3. Check account balance\necho -e \"\\n--- Account ---\"\nvastai show user --raw | python3 -c \"import sys,json; u=json.load(sys.stdin); print(f'Balance: \\${u.get(\\\"balance\\\",0):.2f}')\"\n```\n\n### Incident Type 1: Spot Preemption\n\n**Symptoms**: Instance status changes from `running` to `exited` or `offline` without user action.\n\n```bash\n# 1. Verify preemption (not user error)\nvastai show instance $ID --raw | python3 -c \"\nimport sys, json; i=json.load(sys.stdin)\nprint(f'Status: {i.get(\\\"actual_status\\\")}')\nprint(f'Status msg: {i.get(\\\"status_msg\\\", \\\"none\\\")}')\n\"\n\n# 2. Check if checkpoint was saved\n# (depends on your checkpoint storage — S3, GCS, etc.)\naws s3 ls s3:\u002F\u002Fbucket\u002Fcheckpoints\u002F --recursive | tail -5\n\n# 3. Provision replacement instance\nvastai search offers \"gpu_name=${GPU_NAME} reliability>0.98 rentable=true\" \\\n  --order dph_total --limit 3\n\n# 4. Create replacement and resume from checkpoint\nvastai create instance $NEW_OFFER_ID --image $IMAGE --disk 50\n```\n\n### Incident Type 2: Training Job Crash\n\n**Symptoms**: Instance running but training process exited with error.\n\n```bash\n# 1. SSH in and check logs\nssh -p $PORT root@$HOST \"tail -100 \u002Fworkspace\u002Ftrain.log 2>\u002Fdev\u002Fnull || echo 'No log file'\"\n\n# 2. Common causes\nssh -p $PORT root@$HOST \u003C\u003C 'CHECK'\n# GPU memory issue?\nnvidia-smi | grep -i \"out of memory\" && echo \"OOM detected\"\n# Disk full?\ndf -h \u002Fworkspace | tail -1\n# Process still running?\nps aux | grep python | grep -v grep\nCHECK\n\n# 3. Restart training from checkpoint\nssh -p $PORT root@$HOST \"cd \u002Fworkspace && python train.py --resume-from latest\"\n```\n\n### Incident Type 3: GPU Hardware Failure\n\n**Symptoms**: `nvidia-smi` fails, CUDA errors, or ECC memory errors.\n\n```bash\n# 1. Check GPU health\nssh -p $PORT root@$HOST \"nvidia-smi\" || echo \"GPU not responding\"\n\n# 2. This is a host-level failure — you cannot fix it\n# Destroy the instance and provision on a different host\nvastai destroy instance $ID\n\n# 3. Report the host to Vast.ai support\necho \"Report host ID to Vast.ai support for investigation\"\n```\n\n### Incident Type 4: Billing Emergency\n\n```bash\n# Stop all billing immediately\necho \"EMERGENCY: Destroying all instances\"\nvastai show instances --raw | python3 -c \"\nimport sys, json, subprocess\nfor inst in json.load(sys.stdin):\n    if inst.get('actual_status') in ('running', 'loading'):\n        subprocess.run(['vastai', 'destroy', 'instance', str(inst['id'])])\n        print(f'Destroyed instance {inst[\\\"id\\\"]}')\n\"\n```\n\n### Postmortem Template\n\n```markdown\n## Incident Report\n- **Date**: YYYY-MM-DD\n- **Duration**: X hours\n- **Impact**: N instances affected, $X cost\n- **Root cause**: [spot preemption \u002F OOM \u002F disk full \u002F GPU failure]\n- **Resolution**: [replaced instance \u002F increased VRAM \u002F expanded disk]\n- **Prevention**: [higher reliability filter \u002F checkpoints \u002F auto-recovery]\n```\n\n## Output\n- Triage script with instant status assessment\n- Recovery procedures for 4 incident types\n- Emergency billing stop command\n- Postmortem template\n\n## Error Handling\n| Incident | MTTR Target | Recovery |\n|----------|-------------|----------|\n| Spot preemption | \u003C 10 min | Auto-provision replacement, resume from checkpoint |\n| Training crash | \u003C 5 min | SSH in, diagnose, restart from checkpoint |\n| GPU failure | \u003C 15 min | Destroy instance, provision on different host |\n| Billing emergency | \u003C 1 min | Destroy all instances immediately |\n\n## Resources\n- [Vast.ai Status](https:\u002F\u002Fstatus.vast.ai)\n- [Vast.ai CLI](https:\u002F\u002Fdocs.vast.ai\u002Fcli\u002Fget-started)\n\n## Next Steps\nFor data handling and security, see `vastai-data-handling`.\n\n## Examples\n\n**Auto-recovery script**: Run the event poller from `vastai-webhooks-events` with an auto-recovery handler that provisions a replacement within 5 minutes of preemption.\n\n**Kill switch**: Keep `vastai show instances && vastai destroy instance ALL` aliased for emergency billing stops.\n","# Vast.ai Incident Runbook\n\n## Overview\nRapid incident response procedures for Vast.ai GPU instance failures. Covers triage, mitigation, recovery, and postmortem for common incident types: spot preemption, instance crashes, GPU failures, and billing issues.\n\n## Prerequisites\n- Vast.ai CLI access\n- SSH access to instances (if still running)\n- Checkpoint storage accessible (S3\u002FGCS)\n\n## Instructions\n\n### Triage: Assess Impact (\u003C 2 minutes)\n\n```bash\n#!\u002Fbin\u002Fbash\nset -euo pipefail\necho \"=== INCIDENT TRIAGE ===\"\necho \"Time: $(date -u)\"\n\n# 1. Check all instances\necho -e \"\\n--- Instance Status ---\"\nvastai show instances --raw | python3 -c \"\nimport sys, json\nfor inst in json.load(sys.stdin):\n    status = inst.get('actual_status', '?')\n    flag = 'ALERT' if status in ('error', 'exited', 'offline') else 'OK'\n    print(f'  [{flag}] ID:{inst[\\\"id\\\"]} Status:{status} '\n          f'GPU:{inst.get(\\\"gpu_name\\\",\\\"?\\\")} \\${inst.get(\\\"dph_total\\\",0):.3f}\u002Fhr')\n\"\n\n# 2. Check if affected instance has recent logs\necho -e \"\\n--- Recent Logs (last 20 lines) ---\"\nvastai logs ${INSTANCE_ID:-0} --tail 20 2>\u002Fdev\u002Fnull || echo \"No logs available\"\n\n# 3. Check account balance\necho -e \"\\n--- Account ---\"\nvastai show user --raw | python3 -c \"import sys,json; u=json.load(sys.stdin); print(f'Balance: \\${u.get(\\\"balance\\\",0):.2f}')\"\n```\n\n### Incident Type 1: Spot Preemption\n\n**Symptoms**: Instance status changes from `running` to `exited` or `offline` without user action.\n\n```bash\n# 1. Verify preemption (not user error)\nvastai show instance $ID --raw | python3 -c \"\nimport sys, json; i=json.load(sys.stdin)\nprint(f'Status: {i.get(\\\"actual_status\\\")}')\nprint(f'Status msg: {i.get(\\\"status_msg\\\", \\\"none\\\")}')\n\"\n\n# 2. Check if checkpoint was saved\n# (depends on your checkpoint storage — S3, GCS, etc.)\naws s3 ls s3:\u002F\u002Fbucket\u002Fcheckpoints\u002F --recursive | tail -5\n\n# 3. Provision replacement instance\nvastai search offers \"gpu_name=${GPU_NAME} reliability>0.98 rentable=true\" \\\n  --order dph_total --limit 3\n\n# 4. Create replacement and resume from checkpoint\nvastai create instance $NEW_OFFER_ID --image $IMAGE --disk 50\n```\n\n### Incident Type 2: Training Job Crash\n\n**Symptoms**: Instance running but training process exited with error.\n\n```bash\n# 1. SSH in and check logs\nssh -p $PORT root@$HOST \"tail -100 \u002Fworkspace\u002Ftrain.log 2>\u002Fdev\u002Fnull || echo 'No log file'\"\n\n# 2. Common causes\nssh -p $PORT root@$HOST \u003C\u003C 'CHECK'\n# GPU memory issue?\nnvidia-smi | grep -i \"out of memory\" && echo \"OOM detected\"\n# Disk full?\ndf -h \u002Fworkspace | tail -1\n# Process still running?\nps aux | grep python | grep -v grep\nCHECK\n\n# 3. Restart training from checkpoint\nssh -p $PORT root@$HOST \"cd \u002Fworkspace && python train.py --resume-from latest\"\n```\n\n### Incident Type 3: GPU Hardware Failure\n\n**Symptoms**: `nvidia-smi` fails, CUDA errors, or ECC memory errors.\n\n```bash\n# 1. Check GPU health\nssh -p $PORT root@$HOST \"nvidia-smi\" || echo \"GPU not responding\"\n\n# 2. This is a host-level failure — you cannot fix it\n# Destroy the instance and provision on a different host\nvastai destroy instance $ID\n\n# 3. Report the host to Vast.ai support\necho \"Report host ID to Vast.ai support for investigation\"\n```\n\n### Incident Type 4: Billing Emergency\n\n```bash\n# Stop all billing immediately\necho \"EMERGENCY: Destroying all instances\"\nvastai show instances --raw | python3 -c \"\nimport sys, json, subprocess\nfor inst in json.load(sys.stdin):\n    if inst.get('actual_status') in ('running', 'loading'):\n        subprocess.run(['vastai', 'destroy', 'instance', str(inst['id'])])\n        print(f'Destroyed instance {inst[\\\"id\\\"]}')\n\"\n```\n\n### Postmortem Template\n\n```markdown\n## Incident Report\n- **Date**: YYYY-MM-DD\n- **Duration**: X hours\n- **Impact**: N instances affected, $X cost\n- **Root cause**: [spot preemption \u002F OOM \u002F disk full \u002F GPU failure]\n- **Resolution**: [replaced instance \u002F increased VRAM \u002F expanded disk]\n- **Prevention**: [higher reliability filter \u002F checkpoints \u002F auto-recovery]\n```\n\n## Output\n- Triage script with instant status assessment\n- Recovery procedures for 4 incident types\n- Emergency billing stop command\n- Postmortem template\n\n## Error Handling\n| Incident | MTTR Target | Recovery |\n|----------|-------------|----------|\n| Spot preemption | \u003C 10 min | Auto-provision replacement, resume from checkpoint |\n| Training crash | \u003C 5 min | SSH in, diagnose, restart from checkpoint |\n| GPU failure | \u003C 15 min | Destroy instance, provision on different host |\n| Billing emergency | \u003C 1 min | Destroy all instances immediately |\n\n## Resources\n- [Vast.ai Status](https:\u002F\u002Fstatus.vast.ai)\n- [Vast.ai CLI](https:\u002F\u002Fdocs.vast.ai\u002Fcli\u002Fget-started)\n\n## Next Steps\nFor data handling and security, see `vastai-data-handling`.\n\n## Examples\n\n**Auto-recovery script**: Run the event poller from `vastai-webhooks-events` with an auto-recovery handler that provisions a replacement within 5 minutes of preemption.\n\n**Kill switch**: Keep `vastai show instances && vastai destroy instance ALL` aliased for emergency billing stops.\n",{"name":5,"tags":24,"author":28,"license":15,"version":29,"description":30,"allowed-tools":31,"compatible-with":32},[25,26,27],"saas","vast-ai","incident-response","Jeremy Longshore \u003Cjeremy@intentsolutions.io>","1.0.0","Execute Vast.ai incident response for GPU instance failures and outages.\nUse when responding to instance failures, investigating training crashes,\nor handling spot preemption emergencies.\nTrigger with phrases like \"vastai incident\", \"vastai outage\",\n\"vastai down\", \"vastai emergency\", \"vastai instance failed\".\n","Read, Grep, Bash(vastai:*), Bash(curl:*), Bash(ssh:*)","claude-code, codex, openclaw","main","99825ad3f3bcc6597ddcc7294a14e6e922f2b67b","skills\u002Fsaas-packs--vastai-pack--vastai-incident-runbook",0,"https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills","https:\u002F\u002Fgithub.com\u002Fjukeyman","https:\u002F\u002FRJBUSINESSSOLUTIONS.ORG","Credit and Finance Expert | Developer\r\n\r\nI'm Rick Jefferson, a seasoned credit and finance expert with over a decade of experience in financial literacy,","2026-07-13 07:03:29",{"source":8,"skillName":5,"commands":43,"prompt":53,"zipAvailable":54},[44,47,50],{"tool":45,"command":46},"npx","npx skills add jukeyman\u002Fjukeyman-skills --skill vastai-incident-runbook",{"tool":48,"command":49},"bunx","bunx skills add jukeyman\u002Fjukeyman-skills --skill vastai-incident-runbook",{"tool":51,"command":52},"pnpm","pnpm dlx skills add jukeyman\u002Fjukeyman-skills --skill vastai-incident-runbook","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fsaas-packs--vastai-pack--vastai-incident-runbook\u002FSKILL.md",true,[],[57,64,71,78,85,93],{"skillNo":58,"name":59,"nameZh":6,"description":60,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":61,"categories":62,"keywords":63,"publishedAt":19},"01M79ANS2SM329NEP5ZCX106RR","zustand-store-creator","Create zustand store creator operations. Auto-activating skill for Frontend Development.\nTriggers on: zustand store creator, zustand store creator\nPart of the Frontend Development skill category. Use when working with zustand store creator functionality. Trigger with phrases like \"zustand store creator\", \"zustand creator\", \"zustand\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzustand-store-creator\u002FSKILL.md",[],[],{"skillNo":65,"name":66,"nameZh":6,"description":67,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":68,"categories":69,"keywords":70,"publishedAt":19},"01M79AMZKP4YDEZ7EHG1Y7PP74","zero-trust-config-helper","Configure with zero trust config helper operations. Auto-activating skill for Security Advanced.\nTriggers on: zero trust config helper, zero trust config helper\nPart of the Security Advanced skill category. Use when configuring systems or services. Trigger with phrases like \"zero trust config helper\", \"zero helper\", \"zero\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzero-trust-config-helper\u002FSKILL.md",[],[],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":19},"01M79AMJSBCEJQNBTYNAHYN5HM","zapier-integration-helper","Assist with zapier integration helper operations. Auto-activating skill for Business Automation.\nTriggers on: zapier integration helper, zapier integration helper\nPart of the Business Automation skill category. Use when working with APIs or building integrations. Trigger with phrases like \"zapier integration helper\", \"zapier helper\", \"zapier\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzapier-integration-helper\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":19},"01M79AHJZGVMARS13VFQYRR1MP","yaml-config-validator","Validate yaml config validator operations. Auto-activating skill for DevOps Basics.\nTriggers on: yaml config validator, yaml config validator\nPart of the DevOps Basics skill category. Use when configuring systems or services. Trigger with phrases like \"yaml config validator\", \"yaml validator\", \"yaml\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fyaml-config-validator\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":89,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M79AHGDZRN0020RDQSQDPAM8","xvary-stock-research","Thesis-driven equity analysis from public SEC EDGAR and market data; \u002Fanalyze, \u002Fscore, \u002Fcompare workflows with bundled Python tools (Claude Code, Cursor, Codex).","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxvary-stock-research\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxvary-stock-research\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M79AHCYJ9FV9RMHQ24HDRN33","xss-vulnerability-scanner","Scan xss vulnerability scanner operations. Auto-activating skill for Security Fundamentals.\nTriggers on: xss vulnerability scanner, xss vulnerability scanner\nPart of the Security Fundamentals skill category. Use when working with xss vulnerability scanner functionality. Trigger with phrases like \"xss vulnerability scanner\", \"xss scanner\", \"xss\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxss-vulnerability-scanner\u002FSKILL.md",[],[]]