[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EDRKGXZX4FCAYYKVY1DNT5":3,"skill-same:01M6EDRKGXZX4FCAYYKVY1DNT5":46},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":21,"frontMatter":22,"refName":23,"commitSha":24,"skillDirectory":25,"forksCount":26,"repositoryHtmlUrl":27,"repositoryHomepageUrl":28,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":30,"updatedAt":31,"install":32,"resources":45},"01M6EDRKGXZX4FCAYYKVY1DNT5","nanogpt",null,"ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fnanogpt\u002FSKILL.md",[],[18],"NanoGPT","2026-11-10 10:12:30",false,"# nanoGPT - Minimalist GPT Training\n\n## Quick start\n\nnanoGPT is a simplified GPT implementation designed for learning and experimentation.\n\n**Installation**:\n```bash\npip install torch numpy transformers datasets tiktoken wandb tqdm\n```\n\n**Train on Shakespeare** (CPU-friendly):\n```bash\n# Prepare data\npython data\u002Fshakespeare_char\u002Fprepare.py\n\n# Train (5 minutes on CPU)\npython train.py config\u002Ftrain_shakespeare_char.py\n\n# Generate text\npython sample.py --out_dir=out-shakespeare-char\n```\n\n**Output**:\n```\nROMEO:\nWhat say'st thou? Shall I speak, and be a man?\n\nJULIET:\nI am afeard, and yet I'll speak; for thou art\nOne that hath been a man, and yet I know not\nWhat thou art.\n```\n\n## Common workflows\n\n### Workflow 1: Character-level Shakespeare\n\n**Complete training pipeline**:\n```bash\n# Step 1: Prepare data (creates train.bin, val.bin)\npython data\u002Fshakespeare_char\u002Fprepare.py\n\n# Step 2: Train small model\npython train.py config\u002Ftrain_shakespeare_char.py\n\n# Step 3: Generate text\npython sample.py --out_dir=out-shakespeare-char\n```\n\n**Config** (`config\u002Ftrain_shakespeare_char.py`):\n```python\n# Model config\nn_layer = 6          # 6 transformer layers\nn_head = 6           # 6 attention heads\nn_embd = 384         # 384-dim embeddings\nblock_size = 256     # 256 char context\n\n# Training config\nbatch_size = 64\nlearning_rate = 1e-3\nmax_iters = 5000\neval_interval = 500\n\n# Hardware\ndevice = 'cpu'  # Or 'cuda'\ncompile = False # Set True for PyTorch 2.0\n```\n\n**Training time**: ~5 minutes (CPU), ~1 minute (GPU)\n\n### Workflow 2: Reproduce GPT-2 (124M)\n\n**Multi-GPU training on OpenWebText**:\n```bash\n# Step 1: Prepare OpenWebText (takes ~1 hour)\npython data\u002Fopenwebtext\u002Fprepare.py\n\n# Step 2: Train GPT-2 124M with DDP (8 GPUs)\ntorchrun --standalone --nproc_per_node=8 \\\n  train.py config\u002Ftrain_gpt2.py\n\n# Step 3: Sample from trained model\npython sample.py --out_dir=out\n```\n\n**Config** (`config\u002Ftrain_gpt2.py`):\n```python\n# GPT-2 (124M) architecture\nn_layer = 12\nn_head = 12\nn_embd = 768\nblock_size = 1024\ndropout = 0.0\n\n# Training\nbatch_size = 12\ngradient_accumulation_steps = 5 * 8  # Total batch ~0.5M tokens\nlearning_rate = 6e-4\nmax_iters = 600000\nlr_decay_iters = 600000\n\n# System\ncompile = True  # PyTorch 2.0\n```\n\n**Training time**: ~4 days (8× A100)\n\n### Workflow 3: Fine-tune pretrained GPT-2\n\n**Start from OpenAI checkpoint**:\n```python\n# In train.py or config\ninit_from = 'gpt2'  # Options: gpt2, gpt2-medium, gpt2-large, gpt2-xl\n\n# Model loads OpenAI weights automatically\npython train.py config\u002Ffinetune_shakespeare.py\n```\n\n**Example config** (`config\u002Ffinetune_shakespeare.py`):\n```python\n# Start from GPT-2\ninit_from = 'gpt2'\n\n# Dataset\ndataset = 'shakespeare_char'\nbatch_size = 1\nblock_size = 1024\n\n# Fine-tuning\nlearning_rate = 3e-5  # Lower LR for fine-tuning\nmax_iters = 2000\nwarmup_iters = 100\n\n# Regularization\nweight_decay = 1e-1\n```\n\n### Workflow 4: Custom dataset\n\n**Train on your own text**:\n```python\n# data\u002Fcustom\u002Fprepare.py\nimport numpy as np\n\n# Load your data\nwith open('my_data.txt', 'r') as f:\n    text = f.read()\n\n# Create character mappings\nchars = sorted(list(set(text)))\nstoi = {ch: i for i, ch in enumerate(chars)}\nitos = {i: ch for i, ch in enumerate(chars)}\n\n# Tokenize\ndata = np.array([stoi[ch] for ch in text], dtype=np.uint16)\n\n# Split train\u002Fval\nn = len(data)\ntrain_data = data[:int(n*0.9)]\nval_data = data[int(n*0.9):]\n\n# Save\ntrain_data.tofile('data\u002Fcustom\u002Ftrain.bin')\nval_data.tofile('data\u002Fcustom\u002Fval.bin')\n```\n\n**Train**:\n```bash\npython data\u002Fcustom\u002Fprepare.py\npython train.py --dataset=custom\n```\n\n## When to use vs alternatives\n\n**Use nanoGPT when**:\n- Learning how GPT works\n- Experimenting with transformer variants\n- Teaching\u002Feducation purposes\n- Quick prototyping\n- Limited compute (can run on CPU)\n\n**Simplicity advantages**:\n- **~300 lines**: Entire model in `model.py`\n- **~300 lines**: Training loop in `train.py`\n- **Hackable**: Easy to modify\n- **No abstractions**: Pure PyTorch\n\n**Use alternatives instead**:\n- **HuggingFace Transformers**: Production use, many models\n- **Megatron-LM**: Large-scale distributed training\n- **LitGPT**: More architectures, production-ready\n- **PyTorch Lightning**: Need high-level framework\n\n## Common issues\n\n**Issue: CUDA out of memory**\n\nReduce batch size or context length:\n```python\nbatch_size = 1  # Reduce from 12\nblock_size = 512  # Reduce from 1024\ngradient_accumulation_steps = 40  # Increase to maintain effective batch\n```\n\n**Issue: Training too slow**\n\nEnable compilation (PyTorch 2.0+):\n```python\ncompile = True  # 2× speedup\n```\n\nUse mixed precision:\n```python\ndtype = 'bfloat16'  # Or 'float16'\n```\n\n**Issue: Poor generation quality**\n\nTrain longer:\n```python\nmax_iters = 10000  # Increase from 5000\n```\n\nLower temperature:\n```python\n# In sample.py\ntemperature = 0.7  # Lower from 1.0\ntop_k = 200       # Add top-k sampling\n```\n\n**Issue: Can't load GPT-2 weights**\n\nInstall transformers:\n```bash\npip install transformers\n```\n\nCheck model name:\n```python\ninit_from = 'gpt2'  # Valid: gpt2, gpt2-medium, gpt2-large, gpt2-xl\n```\n\n## Advanced topics\n\n**Model architecture**: See [references\u002Farchitecture.md](references\u002Farchitecture.md) for GPT block structure, multi-head attention, and MLP layers explained simply.\n\n**Training loop**: See [references\u002Ftraining.md](references\u002Ftraining.md) for learning rate schedule, gradient accumulation, and distributed data parallel setup.\n\n**Data preparation**: See [references\u002Fdata.md](references\u002Fdata.md) for tokenization strategies (character-level vs BPE) and binary format details.\n\n## Hardware requirements\n\n- **Shakespeare (char-level)**:\n  - CPU: 5 minutes\n  - GPU (T4): 1 minute\n  - VRAM: \u003C1GB\n\n- **GPT-2 (124M)**:\n  - 1× A100: ~1 week\n  - 8× A100: ~4 days\n  - VRAM: ~16GB per GPU\n\n- **GPT-2 Medium (350M)**:\n  - 8× A100: ~2 weeks\n  - VRAM: ~40GB per GPU\n\n**Performance**:\n- With `compile=True`: 2× speedup\n- With `dtype=bfloat16`: 50% memory reduction\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fkarpathy\u002FnanoGPT ⭐ 48,000+\n- Video: \"Let's build GPT\" by Andrej Karpathy\n- Paper: \"Attention is All You Need\" (Vaswani et al.)\n- OpenWebText: https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FSkylion007\u002Fopenwebtext\n- Educational: Best for understanding transformers from scratch\n",{},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fnanogpt",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":7,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill nanogpt",{"tool":38,"command":39},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill nanogpt",{"tool":41,"command":42},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill nanogpt","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fnanogpt\u002FSKILL.md",true,[],[47,61,71,80,86,93],{"skillNo":48,"name":49,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":50,"categories":51,"keywords":57,"publishedAt":19},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[52],{"categoryNo":53,"categoryName":54,"taxonomyType":55,"taxonomyTypeDesc":56},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[58,59,60],"model","model hierarchy","model hierarchy skill",{"skillNo":62,"name":63,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":67,"publishedAt":19},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[66],{"categoryNo":53,"categoryName":54,"taxonomyType":55,"taxonomyTypeDesc":56},[68,69,70],"MCP","zoom","zoom mcp",{"skillNo":72,"name":73,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[77,78,79,69],"sdk","zoom apps sdk","apps",{"skillNo":81,"name":82,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[82],{"skillNo":87,"name":88,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":92,"publishedAt":19},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[91],{"categoryNo":53,"categoryName":54,"taxonomyType":55,"taxonomyTypeDesc":56},[88],{"skillNo":94,"name":95,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":19},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[95]]