[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M72CW2WQJ1498AD4Z2H88JRH":3,"skill-same:01M72CW2WQJ1498AD4Z2H88JRH":75},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":13,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M72CW2WQJ1498AD4Z2H88JRH","optimizing-attention-flash",null,"Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training\u002Frunning transformers with long sequences (>512 tokens), encountering GPU memory issues with attention, or need faster inference. Supports PyTorch native SDPA, flash-attn library, H100 FP8, and sliding window attention.","Sansar35\u002Fclaude-agent","Sansar35","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F145020281?v=4","01M724CQ2SCC86MGNSD87D0D4K","01M724CQ3965G9WJ3QXE3AJ5YY",1,"MIT","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Foptimizing-attention-flash\u002FSKILL.md",[],[],"2026-11-10 18:39:20",false,"---\nname: optimizing-attention-flash\ndescription: Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training\u002Frunning transformers with long sequences (>512 tokens), encountering GPU memory issues with attention, or need faster inference. Supports PyTorch native SDPA, flash-attn library, H100 FP8, and sliding window attention.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Optimization, Flash Attention, Attention Optimization, Memory Efficiency, Speed Optimization, Long Context, PyTorch, SDPA, H100, FP8, Transformers]\ndependencies: [flash-attn, torch, transformers]\n---\n\n# Flash Attention - Fast Memory-Efficient Attention\n\n## Quick start\n\nFlash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation.\n\n**PyTorch native (easiest, PyTorch 2.2+)**:\n```python\nimport torch\nimport torch.nn.functional as F\n\nq = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)  # [batch, heads, seq, dim]\nk = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\n\n# Automatically uses Flash Attention if available\nout = F.scaled_dot_product_attention(q, k, v)\n```\n\n**flash-attn library (more features)**:\n```bash\npip install flash-attn --no-build-isolation\n```\n\n```python\nfrom flash_attn import flash_attn_func\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nout = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)\n```\n\n## Common workflows\n\n### Workflow 1: Enable in existing PyTorch model\n\nCopy this checklist:\n\n```\nFlash Attention Integration:\n- [ ] Step 1: Check PyTorch version (≥2.2)\n- [ ] Step 2: Enable Flash Attention backend\n- [ ] Step 3: Verify speedup with profiling\n- [ ] Step 4: Test accuracy matches baseline\n```\n\n**Step 1: Check PyTorch version**\n\n```bash\npython -c \"import torch; print(torch.__version__)\"\n# Should be ≥2.2.0\n```\n\nIf \u003C2.2, upgrade:\n```bash\npip install --upgrade torch\n```\n\n**Step 2: Enable Flash Attention backend**\n\nReplace standard attention:\n```python\n# Before (standard attention)\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F math.sqrt(d_k), dim=-1)\nout = attn_weights @ v\n\n# After (Flash Attention)\nimport torch.nn.functional as F\nout = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)\n```\n\nForce Flash Attention backend:\n```python\nwith torch.backends.cuda.sdp_kernel(\n    enable_flash=True,\n    enable_math=False,\n    enable_mem_efficient=False\n):\n    out = F.scaled_dot_product_attention(q, k, v)\n```\n\n**Step 3: Verify speedup with profiling**\n\n```python\nimport torch.utils.benchmark as benchmark\n\ndef test_attention(use_flash):\n    q, k, v = [torch.randn(2, 8, 2048, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n    if use_flash:\n        with torch.backends.cuda.sdp_kernel(enable_flash=True):\n            return F.scaled_dot_product_attention(q, k, v)\n    else:\n        attn = (q @ k.transpose(-2, -1) \u002F 8.0).softmax(dim=-1)\n        return attn @ v\n\n# Benchmark\nt_flash = benchmark.Timer(stmt='test_attention(True)', globals=globals())\nt_standard = benchmark.Timer(stmt='test_attention(False)', globals=globals())\n\nprint(f\"Flash: {t_flash.timeit(100).mean:.3f}s\")\nprint(f\"Standard: {t_standard.timeit(100).mean:.3f}s\")\n```\n\nExpected: 2-4x speedup for sequences >512 tokens.\n\n**Step 4: Test accuracy matches baseline**\n\n```python\n# Compare outputs\nq, k, v = [torch.randn(1, 8, 512, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Flash Attention\nout_flash = F.scaled_dot_product_attention(q, k, v)\n\n# Standard attention\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F 8.0, dim=-1)\nout_standard = attn_weights @ v\n\n# Check difference\ndiff = (out_flash - out_standard).abs().max()\nprint(f\"Max difference: {diff:.6f}\")\n# Should be \u003C1e-3 for float16\n```\n\n### Workflow 2: Use flash-attn library for advanced features\n\nFor multi-query attention, sliding window, or H100 FP8.\n\nCopy this checklist:\n\n```\nflash-attn Library Setup:\n- [ ] Step 1: Install flash-attn library\n- [ ] Step 2: Modify attention code\n- [ ] Step 3: Enable advanced features\n- [ ] Step 4: Benchmark performance\n```\n\n**Step 1: Install flash-attn library**\n\n```bash\n# NVIDIA GPUs (CUDA 12.0+)\npip install flash-attn --no-build-isolation\n\n# Verify installation\npython -c \"from flash_attn import flash_attn_func; print('Success')\"\n```\n\n**Step 2: Modify attention code**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# Input: [batch_size, seq_len, num_heads, head_dim]\n# Transpose from [batch, heads, seq, dim] if needed\nq = q.transpose(1, 2)  # [batch, seq, heads, dim]\nk = k.transpose(1, 2)\nv = v.transpose(1, 2)\n\nout = flash_attn_func(\n    q, k, v,\n    dropout_p=0.1,\n    causal=True,  # For autoregressive models\n    window_size=(-1, -1),  # No sliding window\n    softmax_scale=None  # Auto-scale\n)\n\nout = out.transpose(1, 2)  # Back to [batch, heads, seq, dim]\n```\n\n**Step 3: Enable advanced features**\n\nMulti-query attention (shared K\u002FV across heads):\n```python\nfrom flash_attn import flash_attn_func\n\n# q: [batch, seq, num_q_heads, dim]\n# k, v: [batch, seq, num_kv_heads, dim]  # Fewer KV heads\nout = flash_attn_func(q, k, v)  # Automatically handles MQA\n```\n\nSliding window attention (local attention):\n```python\n# Only attend to window of 256 tokens before\u002Fafter\nout = flash_attn_func(\n    q, k, v,\n    window_size=(256, 256),  # (left, right) window\n    causal=True\n)\n```\n\n**Step 4: Benchmark performance**\n\n```python\nimport torch\nfrom flash_attn import flash_attn_func\nimport time\n\nq, k, v = [torch.randn(4, 4096, 32, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Warmup\nfor _ in range(10):\n    _ = flash_attn_func(q, k, v)\n\n# Benchmark\ntorch.cuda.synchronize()\nstart = time.time()\nfor _ in range(100):\n    out = flash_attn_func(q, k, v)\n    torch.cuda.synchronize()\nend = time.time()\n\nprint(f\"Time per iteration: {(end-start)\u002F100*1000:.2f}ms\")\nprint(f\"Memory allocated: {torch.cuda.max_memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 3: H100 FP8 optimization (FlashAttention-3)\n\nFor maximum performance on H100 GPUs.\n\n```\nFP8 Setup:\n- [ ] Step 1: Verify H100 GPU available\n- [ ] Step 2: Install flash-attn with FP8 support\n- [ ] Step 3: Convert inputs to FP8\n- [ ] Step 4: Run with FP8 attention\n```\n\n**Step 1: Verify H100 GPU**\n\n```bash\nnvidia-smi --query-gpu=name --format=csv\n# Should show \"H100\" or \"H800\"\n```\n\n**Step 2: Install flash-attn with FP8 support**\n\n```bash\npip install flash-attn --no-build-isolation\n# FP8 support included for H100\n```\n\n**Step 3: Convert inputs to FP8**\n\n```python\nimport torch\n\nq = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nk = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\n\n# Convert to float8_e4m3 (FP8)\nq_fp8 = q.to(torch.float8_e4m3fn)\nk_fp8 = k.to(torch.float8_e4m3fn)\nv_fp8 = v.to(torch.float8_e4m3fn)\n```\n\n**Step 4: Run with FP8 attention**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# FlashAttention-3 automatically uses FP8 kernels on H100\nout = flash_attn_func(q_fp8, k_fp8, v_fp8)\n# Result: ~1.2 PFLOPS, 1.5-2x faster than FP16\n```\n\n## When to use vs alternatives\n\n**Use Flash Attention when:**\n- Training transformers with sequences >512 tokens\n- Running inference with long context (>2K tokens)\n- GPU memory constrained (OOM with standard attention)\n- Need 2-4x speedup without accuracy loss\n- Using PyTorch 2.2+ or can install flash-attn\n\n**Use alternatives instead:**\n- **Standard attention**: Sequences \u003C256 tokens (overhead not worth it)\n- **xFormers**: Need more attention variants (not just speed)\n- **Memory-efficient attention**: CPU inference (Flash Attention needs GPU)\n\n## Common issues\n\n**Issue: ImportError: cannot import flash_attn**\n\nInstall with no-build-isolation flag:\n```bash\npip install flash-attn --no-build-isolation\n```\n\nOr install CUDA toolkit first:\n```bash\nconda install cuda -c nvidia\npip install flash-attn --no-build-isolation\n```\n\n**Issue: Slower than expected (no speedup)**\n\nFlash Attention benefits increase with sequence length:\n- \u003C512 tokens: Minimal speedup (10-20%)\n- 512-2K tokens: 2-3x speedup\n- >2K tokens: 3-4x speedup\n\nCheck sequence length is sufficient.\n\n**Issue: RuntimeError: CUDA error**\n\nVerify GPU supports Flash Attention:\n```python\nimport torch\nprint(torch.cuda.get_device_capability())\n# Should be ≥(7, 5) for Turing+\n```\n\nFlash Attention requires:\n- Ampere (A100, A10): ✅ Full support\n- Turing (T4): ✅ Supported\n- Volta (V100): ❌ Not supported\n\n**Issue: Accuracy degradation**\n\nCheck dtype is float16 or bfloat16 (not float32):\n```python\nq = q.to(torch.float16)  # Or torch.bfloat16\n```\n\nFlash Attention uses float16\u002Fbfloat16 for speed. Float32 not supported.\n\n## Advanced topics\n\n**Integration with HuggingFace Transformers**: See [references\u002Ftransformers-integration.md](references\u002Ftransformers-integration.md) for enabling Flash Attention in BERT, GPT, Llama models.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for detailed speed and memory comparisons across GPUs and sequence lengths.\n\n**Algorithm details**: See [references\u002Falgorithm.md](references\u002Falgorithm.md) for tiling strategy, recomputation, and IO complexity analysis.\n\n**Advanced features**: See [references\u002Fadvanced-features.md](references\u002Fadvanced-features.md) for rotary embeddings, ALiBi, paged KV cache, and custom attention masks.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, A10, A30) or AMD MI200+\n- **VRAM**: Same as standard attention (Flash Attention doesn't increase memory)\n- **CUDA**: 12.0+ (11.8 minimum)\n- **PyTorch**: 2.2+ for native support\n\n**Not supported**: V100 (Volta), CPU inference\n\n## Resources\n\n- Paper: \"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\" (NeurIPS 2022)\n- Paper: \"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning\" (ICLR 2024)\n- Blog: https:\u002F\u002Ftridao.me\u002Fblog\u002F2024\u002Fflash3\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention\n- PyTorch docs: https:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Fgenerated\u002Ftorch.nn.functional.scaled_dot_product_attention.html\n\n\n\n","# Flash Attention - Fast Memory-Efficient Attention\n\n## Quick start\n\nFlash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation.\n\n**PyTorch native (easiest, PyTorch 2.2+)**:\n```python\nimport torch\nimport torch.nn.functional as F\n\nq = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)  # [batch, heads, seq, dim]\nk = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\n\n# Automatically uses Flash Attention if available\nout = F.scaled_dot_product_attention(q, k, v)\n```\n\n**flash-attn library (more features)**:\n```bash\npip install flash-attn --no-build-isolation\n```\n\n```python\nfrom flash_attn import flash_attn_func\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nout = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)\n```\n\n## Common workflows\n\n### Workflow 1: Enable in existing PyTorch model\n\nCopy this checklist:\n\n```\nFlash Attention Integration:\n- [ ] Step 1: Check PyTorch version (≥2.2)\n- [ ] Step 2: Enable Flash Attention backend\n- [ ] Step 3: Verify speedup with profiling\n- [ ] Step 4: Test accuracy matches baseline\n```\n\n**Step 1: Check PyTorch version**\n\n```bash\npython -c \"import torch; print(torch.__version__)\"\n# Should be ≥2.2.0\n```\n\nIf \u003C2.2, upgrade:\n```bash\npip install --upgrade torch\n```\n\n**Step 2: Enable Flash Attention backend**\n\nReplace standard attention:\n```python\n# Before (standard attention)\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F math.sqrt(d_k), dim=-1)\nout = attn_weights @ v\n\n# After (Flash Attention)\nimport torch.nn.functional as F\nout = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)\n```\n\nForce Flash Attention backend:\n```python\nwith torch.backends.cuda.sdp_kernel(\n    enable_flash=True,\n    enable_math=False,\n    enable_mem_efficient=False\n):\n    out = F.scaled_dot_product_attention(q, k, v)\n```\n\n**Step 3: Verify speedup with profiling**\n\n```python\nimport torch.utils.benchmark as benchmark\n\ndef test_attention(use_flash):\n    q, k, v = [torch.randn(2, 8, 2048, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n    if use_flash:\n        with torch.backends.cuda.sdp_kernel(enable_flash=True):\n            return F.scaled_dot_product_attention(q, k, v)\n    else:\n        attn = (q @ k.transpose(-2, -1) \u002F 8.0).softmax(dim=-1)\n        return attn @ v\n\n# Benchmark\nt_flash = benchmark.Timer(stmt='test_attention(True)', globals=globals())\nt_standard = benchmark.Timer(stmt='test_attention(False)', globals=globals())\n\nprint(f\"Flash: {t_flash.timeit(100).mean:.3f}s\")\nprint(f\"Standard: {t_standard.timeit(100).mean:.3f}s\")\n```\n\nExpected: 2-4x speedup for sequences >512 tokens.\n\n**Step 4: Test accuracy matches baseline**\n\n```python\n# Compare outputs\nq, k, v = [torch.randn(1, 8, 512, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Flash Attention\nout_flash = F.scaled_dot_product_attention(q, k, v)\n\n# Standard attention\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F 8.0, dim=-1)\nout_standard = attn_weights @ v\n\n# Check difference\ndiff = (out_flash - out_standard).abs().max()\nprint(f\"Max difference: {diff:.6f}\")\n# Should be \u003C1e-3 for float16\n```\n\n### Workflow 2: Use flash-attn library for advanced features\n\nFor multi-query attention, sliding window, or H100 FP8.\n\nCopy this checklist:\n\n```\nflash-attn Library Setup:\n- [ ] Step 1: Install flash-attn library\n- [ ] Step 2: Modify attention code\n- [ ] Step 3: Enable advanced features\n- [ ] Step 4: Benchmark performance\n```\n\n**Step 1: Install flash-attn library**\n\n```bash\n# NVIDIA GPUs (CUDA 12.0+)\npip install flash-attn --no-build-isolation\n\n# Verify installation\npython -c \"from flash_attn import flash_attn_func; print('Success')\"\n```\n\n**Step 2: Modify attention code**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# Input: [batch_size, seq_len, num_heads, head_dim]\n# Transpose from [batch, heads, seq, dim] if needed\nq = q.transpose(1, 2)  # [batch, seq, heads, dim]\nk = k.transpose(1, 2)\nv = v.transpose(1, 2)\n\nout = flash_attn_func(\n    q, k, v,\n    dropout_p=0.1,\n    causal=True,  # For autoregressive models\n    window_size=(-1, -1),  # No sliding window\n    softmax_scale=None  # Auto-scale\n)\n\nout = out.transpose(1, 2)  # Back to [batch, heads, seq, dim]\n```\n\n**Step 3: Enable advanced features**\n\nMulti-query attention (shared K\u002FV across heads):\n```python\nfrom flash_attn import flash_attn_func\n\n# q: [batch, seq, num_q_heads, dim]\n# k, v: [batch, seq, num_kv_heads, dim]  # Fewer KV heads\nout = flash_attn_func(q, k, v)  # Automatically handles MQA\n```\n\nSliding window attention (local attention):\n```python\n# Only attend to window of 256 tokens before\u002Fafter\nout = flash_attn_func(\n    q, k, v,\n    window_size=(256, 256),  # (left, right) window\n    causal=True\n)\n```\n\n**Step 4: Benchmark performance**\n\n```python\nimport torch\nfrom flash_attn import flash_attn_func\nimport time\n\nq, k, v = [torch.randn(4, 4096, 32, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Warmup\nfor _ in range(10):\n    _ = flash_attn_func(q, k, v)\n\n# Benchmark\ntorch.cuda.synchronize()\nstart = time.time()\nfor _ in range(100):\n    out = flash_attn_func(q, k, v)\n    torch.cuda.synchronize()\nend = time.time()\n\nprint(f\"Time per iteration: {(end-start)\u002F100*1000:.2f}ms\")\nprint(f\"Memory allocated: {torch.cuda.max_memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 3: H100 FP8 optimization (FlashAttention-3)\n\nFor maximum performance on H100 GPUs.\n\n```\nFP8 Setup:\n- [ ] Step 1: Verify H100 GPU available\n- [ ] Step 2: Install flash-attn with FP8 support\n- [ ] Step 3: Convert inputs to FP8\n- [ ] Step 4: Run with FP8 attention\n```\n\n**Step 1: Verify H100 GPU**\n\n```bash\nnvidia-smi --query-gpu=name --format=csv\n# Should show \"H100\" or \"H800\"\n```\n\n**Step 2: Install flash-attn with FP8 support**\n\n```bash\npip install flash-attn --no-build-isolation\n# FP8 support included for H100\n```\n\n**Step 3: Convert inputs to FP8**\n\n```python\nimport torch\n\nq = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nk = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\n\n# Convert to float8_e4m3 (FP8)\nq_fp8 = q.to(torch.float8_e4m3fn)\nk_fp8 = k.to(torch.float8_e4m3fn)\nv_fp8 = v.to(torch.float8_e4m3fn)\n```\n\n**Step 4: Run with FP8 attention**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# FlashAttention-3 automatically uses FP8 kernels on H100\nout = flash_attn_func(q_fp8, k_fp8, v_fp8)\n# Result: ~1.2 PFLOPS, 1.5-2x faster than FP16\n```\n\n## When to use vs alternatives\n\n**Use Flash Attention when:**\n- Training transformers with sequences >512 tokens\n- Running inference with long context (>2K tokens)\n- GPU memory constrained (OOM with standard attention)\n- Need 2-4x speedup without accuracy loss\n- Using PyTorch 2.2+ or can install flash-attn\n\n**Use alternatives instead:**\n- **Standard attention**: Sequences \u003C256 tokens (overhead not worth it)\n- **xFormers**: Need more attention variants (not just speed)\n- **Memory-efficient attention**: CPU inference (Flash Attention needs GPU)\n\n## Common issues\n\n**Issue: ImportError: cannot import flash_attn**\n\nInstall with no-build-isolation flag:\n```bash\npip install flash-attn --no-build-isolation\n```\n\nOr install CUDA toolkit first:\n```bash\nconda install cuda -c nvidia\npip install flash-attn --no-build-isolation\n```\n\n**Issue: Slower than expected (no speedup)**\n\nFlash Attention benefits increase with sequence length:\n- \u003C512 tokens: Minimal speedup (10-20%)\n- 512-2K tokens: 2-3x speedup\n- >2K tokens: 3-4x speedup\n\nCheck sequence length is sufficient.\n\n**Issue: RuntimeError: CUDA error**\n\nVerify GPU supports Flash Attention:\n```python\nimport torch\nprint(torch.cuda.get_device_capability())\n# Should be ≥(7, 5) for Turing+\n```\n\nFlash Attention requires:\n- Ampere (A100, A10): ✅ Full support\n- Turing (T4): ✅ Supported\n- Volta (V100): ❌ Not supported\n\n**Issue: Accuracy degradation**\n\nCheck dtype is float16 or bfloat16 (not float32):\n```python\nq = q.to(torch.float16)  # Or torch.bfloat16\n```\n\nFlash Attention uses float16\u002Fbfloat16 for speed. Float32 not supported.\n\n## Advanced topics\n\n**Integration with HuggingFace Transformers**: See [references\u002Ftransformers-integration.md](references\u002Ftransformers-integration.md) for enabling Flash Attention in BERT, GPT, Llama models.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for detailed speed and memory comparisons across GPUs and sequence lengths.\n\n**Algorithm details**: See [references\u002Falgorithm.md](references\u002Falgorithm.md) for tiling strategy, recomputation, and IO complexity analysis.\n\n**Advanced features**: See [references\u002Fadvanced-features.md](references\u002Fadvanced-features.md) for rotary embeddings, ALiBi, paged KV cache, and custom attention masks.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, A10, A30) or AMD MI200+\n- **VRAM**: Same as standard attention (Flash Attention doesn't increase memory)\n- **CUDA**: 12.0+ (11.8 minimum)\n- **PyTorch**: 2.2+ for native support\n\n**Not supported**: V100 (Volta), CPU inference\n\n## Resources\n\n- Paper: \"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\" (NeurIPS 2022)\n- Paper: \"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning\" (ICLR 2024)\n- Blog: https:\u002F\u002Ftridao.me\u002Fblog\u002F2024\u002Fflash3\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention\n- PyTorch docs: https:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Fgenerated\u002Ftorch.nn.functional.scaled_dot_product_attention.html\n\n\n\n",{"name":5,"tags":23,"author":35,"license":14,"version":36,"description":7,"dependencies":37},[24,25,26,27,28,29,30,31,32,33,34],"Optimization","Flash Attention","Attention Optimization","Memory Efficiency","Speed Optimization","Long Context","PyTorch","SDPA","H100","FP8","Transformers","Orchestra Research","1.0.0",[38,39,40],"flash-attn","torch","transformers","main","d046d088ce5a79a083f0abc0fe1b7ded73bc5b1f","claude-config\u002Fskills\u002Foptimizing-attention-flash","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent","https:\u002F\u002Fgithub.com\u002FSansar35","2026-05-10 16:04:53",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add Sansar35\u002Fclaude-agent --skill optimizing-attention-flash",{"tool":53,"command":54},"bunx","bunx skills add Sansar35\u002Fclaude-agent --skill optimizing-attention-flash",{"tool":56,"command":57},"pnpm","pnpm dlx skills add Sansar35\u002Fclaude-agent --skill optimizing-attention-flash","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Foptimizing-attention-flash\u002FSKILL.md",true,[61,67,71],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},".skill-source.json","REFERENCE","参考文档",595,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Foptimizing-attention-flash\u002F.skill-source.json",{"path":68,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Fbenchmarks.md",7129,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Foptimizing-attention-flash\u002Freferences\u002Fbenchmarks.md",{"path":72,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Ftransformers-integration.md",7427,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Foptimizing-attention-flash\u002Freferences\u002Ftransformers-integration.md",[76,83,90,97,104,112],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":18},"01M72DVAN1JYEA302MT0C5MGD6","zustand-store-ts","Create Zustand stores following established patterns with proper TypeScript types and middleware.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzustand-store-ts\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":18},"01M72DV8J5NBRRM4RPDFRV95N7","zod-validation-expert","Expert in Zod — TypeScript-first schema validation. Covers parsing, custom errors, refinements, type inference, and integration with React Hook Form, Next.js, and tRPC.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzod-validation-expert\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":18},"01M72DV5EMHW0BHP9BZ167MDCB","zinc-database","Access ZINC (230M+ purchasable compounds). Search by ZINC ID\u002FSMILES, similarity searches, 3D-ready structures for docking, analog discovery, for virtual screening and drug discovery.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzinc-database\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":18},"01M72DV216QY0E693ZPEVQ0DBS","zarr-python","Chunked N-D arrays for cloud storage. Compressed arrays, parallel I\u002FO, S3\u002FGCS integration, NumPy\u002FDask\u002FXarray compatible, for large-scale scientific computing pipelines.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":108,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":18},"01M72DTXQREMCRKA8NHCVKHN4W","zapier-workflows","Manage and trigger pre-built Zapier workflows and MCP tool orchestration. Use when user mentions workflows, Zaps, automations, daily digest, research, search, lead tracking, expenses, or asks to \"run\" any process. Also handles Perplexity-based research and Google Sheets data tracking.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-workflows\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-workflows\u002FSKILL.md",[],[],{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":116,"categories":117,"keywords":118,"publishedAt":18},"01M72DTT17TBSJ0Y8J2XZ4C9TZ","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-make-patterns\u002FSKILL.md",[],[]]