[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7AYK0EM8M28XZ9KB7W1XDN3":3,"skill-same:01M7AYK0EM8M28XZ9KB7W1XDN3":54},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":14,"repositoryHtmlUrl":27,"repositoryHomepageUrl":6,"ownerHtmlUrl":28,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":29,"install":30,"resources":43},"01M7AYK0EM8M28XZ9KB7W1XDN3","optimizing-attention-flash",null,"Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training\u002Frunning transformers with long sequences (>512 tokens), encountering GPU memory issues with attention, or need faster inference. Supports PyTorch native SDPA, flash-attn library, H100 FP8, and sliding window attention.","SiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot","SiddhamShah972","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F98964854?v=4","01M7AWVFA8QBCFE7PRDT5XBB2E","Siddham Shah","01M7AWVFB82AN6H6BNQSWEF402",0,"MIT","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Foptimizing-attention-flash\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\ndescription: \"Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training\u002Frunning transformers with long sequences (>512 tokens), encountering GPU memory issues with attention, or need faster inference. Supports PyTorch native SDPA, flash-attn library, H100 FP8, and sliding window attention.\"\nlicense: \"MIT\"\n---\n# Flash Attention - Fast Memory-Efficient Attention\n\n## Quick start\n\nFlash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation.\n\n**PyTorch native (easiest, PyTorch 2.2+)**:\n```python\nimport torch\nimport torch.nn.functional as F\n\nq = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)  # [batch, heads, seq, dim]\nk = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\n\n# Automatically uses Flash Attention if available\nout = F.scaled_dot_product_attention(q, k, v)\n```\n\n**flash-attn library (more features)**:\n```bash\npip install flash-attn --no-build-isolation\n```\n\n```python\nfrom flash_attn import flash_attn_func\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nout = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)\n```\n\n## Common workflows\n\n### Workflow 1: Enable in existing PyTorch model\n\nCopy this checklist:\n\n```\nFlash Attention Integration:\n- [ ] Step 1: Check PyTorch version (≥2.2)\n- [ ] Step 2: Enable Flash Attention backend\n- [ ] Step 3: Verify speedup with profiling\n- [ ] Step 4: Test accuracy matches baseline\n```\n\n**Step 1: Check PyTorch version**\n\n```bash\npython -c \"import torch; print(torch.__version__)\"\n# Should be ≥2.2.0\n```\n\nIf \u003C2.2, upgrade:\n```bash\npip install --upgrade torch\n```\n\n**Step 2: Enable Flash Attention backend**\n\nReplace standard attention:\n```python\n# Before (standard attention)\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F math.sqrt(d_k), dim=-1)\nout = attn_weights @ v\n\n# After (Flash Attention)\nimport torch.nn.functional as F\nout = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)\n```\n\nForce Flash Attention backend:\n```python\nwith torch.backends.cuda.sdp_kernel(\n    enable_flash=True,\n    enable_math=False,\n    enable_mem_efficient=False\n):\n    out = F.scaled_dot_product_attention(q, k, v)\n```\n\n**Step 3: Verify speedup with profiling**\n\n```python\nimport torch.utils.benchmark as benchmark\n\ndef test_attention(use_flash):\n    q, k, v = [torch.randn(2, 8, 2048, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n    if use_flash:\n        with torch.backends.cuda.sdp_kernel(enable_flash=True):\n            return F.scaled_dot_product_attention(q, k, v)\n    else:\n        attn = (q @ k.transpose(-2, -1) \u002F 8.0).softmax(dim=-1)\n        return attn @ v\n\n# Benchmark\nt_flash = benchmark.Timer(stmt='test_attention(True)', globals=globals())\nt_standard = benchmark.Timer(stmt='test_attention(False)', globals=globals())\n\nprint(f\"Flash: {t_flash.timeit(100).mean:.3f}s\")\nprint(f\"Standard: {t_standard.timeit(100).mean:.3f}s\")\n```\n\nExpected: 2-4x speedup for sequences >512 tokens.\n\n**Step 4: Test accuracy matches baseline**\n\n```python\n# Compare outputs\nq, k, v = [torch.randn(1, 8, 512, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Flash Attention\nout_flash = F.scaled_dot_product_attention(q, k, v)\n\n# Standard attention\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F 8.0, dim=-1)\nout_standard = attn_weights @ v\n\n# Check difference\ndiff = (out_flash - out_standard).abs().max()\nprint(f\"Max difference: {diff:.6f}\")\n# Should be \u003C1e-3 for float16\n```\n\n### Workflow 2: Use flash-attn library for advanced features\n\nFor multi-query attention, sliding window, or H100 FP8.\n\nCopy this checklist:\n\n```\nflash-attn Library Setup:\n- [ ] Step 1: Install flash-attn library\n- [ ] Step 2: Modify attention code\n- [ ] Step 3: Enable advanced features\n- [ ] Step 4: Benchmark performance\n```\n\n**Step 1: Install flash-attn library**\n\n```bash\n# NVIDIA GPUs (CUDA 12.0+)\npip install flash-attn --no-build-isolation\n\n# Verify installation\npython -c \"from flash_attn import flash_attn_func; print('Success')\"\n```\n\n**Step 2: Modify attention code**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# Input: [batch_size, seq_len, num_heads, head_dim]\n# Transpose from [batch, heads, seq, dim] if needed\nq = q.transpose(1, 2)  # [batch, seq, heads, dim]\nk = k.transpose(1, 2)\nv = v.transpose(1, 2)\n\nout = flash_attn_func(\n    q, k, v,\n    dropout_p=0.1,\n    causal=True,  # For autoregressive models\n    window_size=(-1, -1),  # No sliding window\n    softmax_scale=None  # Auto-scale\n)\n\nout = out.transpose(1, 2)  # Back to [batch, heads, seq, dim]\n```\n\n**Step 3: Enable advanced features**\n\nMulti-query attention (shared K\u002FV across heads):\n```python\nfrom flash_attn import flash_attn_func\n\n# q: [batch, seq, num_q_heads, dim]\n# k, v: [batch, seq, num_kv_heads, dim]  # Fewer KV heads\nout = flash_attn_func(q, k, v)  # Automatically handles MQA\n```\n\nSliding window attention (local attention):\n```python\n# Only attend to window of 256 tokens before\u002Fafter\nout = flash_attn_func(\n    q, k, v,\n    window_size=(256, 256),  # (left, right) window\n    causal=True\n)\n```\n\n**Step 4: Benchmark performance**\n\n```python\nimport torch\nfrom flash_attn import flash_attn_func\nimport time\n\nq, k, v = [torch.randn(4, 4096, 32, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Warmup\nfor _ in range(10):\n    _ = flash_attn_func(q, k, v)\n\n# Benchmark\ntorch.cuda.synchronize()\nstart = time.time()\nfor _ in range(100):\n    out = flash_attn_func(q, k, v)\n    torch.cuda.synchronize()\nend = time.time()\n\nprint(f\"Time per iteration: {(end-start)\u002F100*1000:.2f}ms\")\nprint(f\"Memory allocated: {torch.cuda.max_memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 3: H100 FP8 optimization (FlashAttention-3)\n\nFor maximum performance on H100 GPUs.\n\n```\nFP8 Setup:\n- [ ] Step 1: Verify H100 GPU available\n- [ ] Step 2: Install flash-attn with FP8 support\n- [ ] Step 3: Convert inputs to FP8\n- [ ] Step 4: Run with FP8 attention\n```\n\n**Step 1: Verify H100 GPU**\n\n```bash\nnvidia-smi --query-gpu=name --format=csv\n# Should show \"H100\" or \"H800\"\n```\n\n**Step 2: Install flash-attn with FP8 support**\n\n```bash\npip install flash-attn --no-build-isolation\n# FP8 support included for H100\n```\n\n**Step 3: Convert inputs to FP8**\n\n```python\nimport torch\n\nq = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nk = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\n\n# Convert to float8_e4m3 (FP8)\nq_fp8 = q.to(torch.float8_e4m3fn)\nk_fp8 = k.to(torch.float8_e4m3fn)\nv_fp8 = v.to(torch.float8_e4m3fn)\n```\n\n**Step 4: Run with FP8 attention**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# FlashAttention-3 automatically uses FP8 kernels on H100\nout = flash_attn_func(q_fp8, k_fp8, v_fp8)\n# Result: ~1.2 PFLOPS, 1.5-2x faster than FP16\n```\n\n## When to use vs alternatives\n\n**Use Flash Attention when:**\n- Training transformers with sequences >512 tokens\n- Running inference with long context (>2K tokens)\n- GPU memory constrained (OOM with standard attention)\n- Need 2-4x speedup without accuracy loss\n- Using PyTorch 2.2+ or can install flash-attn\n\n**Use alternatives instead:**\n- **Standard attention**: Sequences \u003C256 tokens (overhead not worth it)\n- **xFormers**: Need more attention variants (not just speed)\n- **Memory-efficient attention**: CPU inference (Flash Attention needs GPU)\n\n## Common issues\n\n**Issue: ImportError: cannot import flash_attn**\n\nInstall with no-build-isolation flag:\n```bash\npip install flash-attn --no-build-isolation\n```\n\nOr install CUDA toolkit first:\n```bash\nconda install cuda -c nvidia\npip install flash-attn --no-build-isolation\n```\n\n**Issue: Slower than expected (no speedup)**\n\nFlash Attention benefits increase with sequence length:\n- \u003C512 tokens: Minimal speedup (10-20%)\n- 512-2K tokens: 2-3x speedup\n- >2K tokens: 3-4x speedup\n\nCheck sequence length is sufficient.\n\n**Issue: RuntimeError: CUDA error**\n\nVerify GPU supports Flash Attention:\n```python\nimport torch\nprint(torch.cuda.get_device_capability())\n# Should be ≥(7, 5) for Turing+\n```\n\nFlash Attention requires:\n- Ampere (A100, A10): ✅ Full support\n- Turing (T4): ✅ Supported\n- Volta (V100): ❌ Not supported\n\n**Issue: Accuracy degradation**\n\nCheck dtype is float16 or bfloat16 (not float32):\n```python\nq = q.to(torch.float16)  # Or torch.bfloat16\n```\n\nFlash Attention uses float16\u002Fbfloat16 for speed. Float32 not supported.\n\n## Advanced topics\n\n**Integration with HuggingFace Transformers**: See [references\u002Ftransformers-integration.md](references\u002Ftransformers-integration.md) for enabling Flash Attention in BERT, GPT, Llama models.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for detailed speed and memory comparisons across GPUs and sequence lengths.\n\n**Algorithm details**: See [references\u002Falgorithm.md](references\u002Falgorithm.md) for tiling strategy, recomputation, and IO complexity analysis.\n\n**Advanced features**: See [references\u002Fadvanced-features.md](references\u002Fadvanced-features.md) for rotary embeddings, ALiBi, paged KV cache, and custom attention masks.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, A10, A30) or AMD MI200+\n- **VRAM**: Same as standard attention (Flash Attention doesn't increase memory)\n- **CUDA**: 12.0+ (11.8 minimum)\n- **PyTorch**: 2.2+ for native support\n\n**Not supported**: V100 (Volta), CPU inference\n\n## Resources\n\n- Paper: \"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\" (NeurIPS 2022)\n- Paper: \"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning\" (ICLR 2024)\n- Blog: https:\u002F\u002Ftridao.me\u002Fblog\u002F2024\u002Fflash3\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention\n- PyTorch docs: https:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Fgenerated\u002Ftorch.nn.functional.scaled_dot_product_attention.html\n\n\n\n","# Flash Attention - Fast Memory-Efficient Attention\n\n## Quick start\n\nFlash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation.\n\n**PyTorch native (easiest, PyTorch 2.2+)**:\n```python\nimport torch\nimport torch.nn.functional as F\n\nq = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)  # [batch, heads, seq, dim]\nk = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\n\n# Automatically uses Flash Attention if available\nout = F.scaled_dot_product_attention(q, k, v)\n```\n\n**flash-attn library (more features)**:\n```bash\npip install flash-attn --no-build-isolation\n```\n\n```python\nfrom flash_attn import flash_attn_func\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nout = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)\n```\n\n## Common workflows\n\n### Workflow 1: Enable in existing PyTorch model\n\nCopy this checklist:\n\n```\nFlash Attention Integration:\n- [ ] Step 1: Check PyTorch version (≥2.2)\n- [ ] Step 2: Enable Flash Attention backend\n- [ ] Step 3: Verify speedup with profiling\n- [ ] Step 4: Test accuracy matches baseline\n```\n\n**Step 1: Check PyTorch version**\n\n```bash\npython -c \"import torch; print(torch.__version__)\"\n# Should be ≥2.2.0\n```\n\nIf \u003C2.2, upgrade:\n```bash\npip install --upgrade torch\n```\n\n**Step 2: Enable Flash Attention backend**\n\nReplace standard attention:\n```python\n# Before (standard attention)\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F math.sqrt(d_k), dim=-1)\nout = attn_weights @ v\n\n# After (Flash Attention)\nimport torch.nn.functional as F\nout = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)\n```\n\nForce Flash Attention backend:\n```python\nwith torch.backends.cuda.sdp_kernel(\n    enable_flash=True,\n    enable_math=False,\n    enable_mem_efficient=False\n):\n    out = F.scaled_dot_product_attention(q, k, v)\n```\n\n**Step 3: Verify speedup with profiling**\n\n```python\nimport torch.utils.benchmark as benchmark\n\ndef test_attention(use_flash):\n    q, k, v = [torch.randn(2, 8, 2048, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n    if use_flash:\n        with torch.backends.cuda.sdp_kernel(enable_flash=True):\n            return F.scaled_dot_product_attention(q, k, v)\n    else:\n        attn = (q @ k.transpose(-2, -1) \u002F 8.0).softmax(dim=-1)\n        return attn @ v\n\n# Benchmark\nt_flash = benchmark.Timer(stmt='test_attention(True)', globals=globals())\nt_standard = benchmark.Timer(stmt='test_attention(False)', globals=globals())\n\nprint(f\"Flash: {t_flash.timeit(100).mean:.3f}s\")\nprint(f\"Standard: {t_standard.timeit(100).mean:.3f}s\")\n```\n\nExpected: 2-4x speedup for sequences >512 tokens.\n\n**Step 4: Test accuracy matches baseline**\n\n```python\n# Compare outputs\nq, k, v = [torch.randn(1, 8, 512, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Flash Attention\nout_flash = F.scaled_dot_product_attention(q, k, v)\n\n# Standard attention\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F 8.0, dim=-1)\nout_standard = attn_weights @ v\n\n# Check difference\ndiff = (out_flash - out_standard).abs().max()\nprint(f\"Max difference: {diff:.6f}\")\n# Should be \u003C1e-3 for float16\n```\n\n### Workflow 2: Use flash-attn library for advanced features\n\nFor multi-query attention, sliding window, or H100 FP8.\n\nCopy this checklist:\n\n```\nflash-attn Library Setup:\n- [ ] Step 1: Install flash-attn library\n- [ ] Step 2: Modify attention code\n- [ ] Step 3: Enable advanced features\n- [ ] Step 4: Benchmark performance\n```\n\n**Step 1: Install flash-attn library**\n\n```bash\n# NVIDIA GPUs (CUDA 12.0+)\npip install flash-attn --no-build-isolation\n\n# Verify installation\npython -c \"from flash_attn import flash_attn_func; print('Success')\"\n```\n\n**Step 2: Modify attention code**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# Input: [batch_size, seq_len, num_heads, head_dim]\n# Transpose from [batch, heads, seq, dim] if needed\nq = q.transpose(1, 2)  # [batch, seq, heads, dim]\nk = k.transpose(1, 2)\nv = v.transpose(1, 2)\n\nout = flash_attn_func(\n    q, k, v,\n    dropout_p=0.1,\n    causal=True,  # For autoregressive models\n    window_size=(-1, -1),  # No sliding window\n    softmax_scale=None  # Auto-scale\n)\n\nout = out.transpose(1, 2)  # Back to [batch, heads, seq, dim]\n```\n\n**Step 3: Enable advanced features**\n\nMulti-query attention (shared K\u002FV across heads):\n```python\nfrom flash_attn import flash_attn_func\n\n# q: [batch, seq, num_q_heads, dim]\n# k, v: [batch, seq, num_kv_heads, dim]  # Fewer KV heads\nout = flash_attn_func(q, k, v)  # Automatically handles MQA\n```\n\nSliding window attention (local attention):\n```python\n# Only attend to window of 256 tokens before\u002Fafter\nout = flash_attn_func(\n    q, k, v,\n    window_size=(256, 256),  # (left, right) window\n    causal=True\n)\n```\n\n**Step 4: Benchmark performance**\n\n```python\nimport torch\nfrom flash_attn import flash_attn_func\nimport time\n\nq, k, v = [torch.randn(4, 4096, 32, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Warmup\nfor _ in range(10):\n    _ = flash_attn_func(q, k, v)\n\n# Benchmark\ntorch.cuda.synchronize()\nstart = time.time()\nfor _ in range(100):\n    out = flash_attn_func(q, k, v)\n    torch.cuda.synchronize()\nend = time.time()\n\nprint(f\"Time per iteration: {(end-start)\u002F100*1000:.2f}ms\")\nprint(f\"Memory allocated: {torch.cuda.max_memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 3: H100 FP8 optimization (FlashAttention-3)\n\nFor maximum performance on H100 GPUs.\n\n```\nFP8 Setup:\n- [ ] Step 1: Verify H100 GPU available\n- [ ] Step 2: Install flash-attn with FP8 support\n- [ ] Step 3: Convert inputs to FP8\n- [ ] Step 4: Run with FP8 attention\n```\n\n**Step 1: Verify H100 GPU**\n\n```bash\nnvidia-smi --query-gpu=name --format=csv\n# Should show \"H100\" or \"H800\"\n```\n\n**Step 2: Install flash-attn with FP8 support**\n\n```bash\npip install flash-attn --no-build-isolation\n# FP8 support included for H100\n```\n\n**Step 3: Convert inputs to FP8**\n\n```python\nimport torch\n\nq = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nk = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\n\n# Convert to float8_e4m3 (FP8)\nq_fp8 = q.to(torch.float8_e4m3fn)\nk_fp8 = k.to(torch.float8_e4m3fn)\nv_fp8 = v.to(torch.float8_e4m3fn)\n```\n\n**Step 4: Run with FP8 attention**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# FlashAttention-3 automatically uses FP8 kernels on H100\nout = flash_attn_func(q_fp8, k_fp8, v_fp8)\n# Result: ~1.2 PFLOPS, 1.5-2x faster than FP16\n```\n\n## When to use vs alternatives\n\n**Use Flash Attention when:**\n- Training transformers with sequences >512 tokens\n- Running inference with long context (>2K tokens)\n- GPU memory constrained (OOM with standard attention)\n- Need 2-4x speedup without accuracy loss\n- Using PyTorch 2.2+ or can install flash-attn\n\n**Use alternatives instead:**\n- **Standard attention**: Sequences \u003C256 tokens (overhead not worth it)\n- **xFormers**: Need more attention variants (not just speed)\n- **Memory-efficient attention**: CPU inference (Flash Attention needs GPU)\n\n## Common issues\n\n**Issue: ImportError: cannot import flash_attn**\n\nInstall with no-build-isolation flag:\n```bash\npip install flash-attn --no-build-isolation\n```\n\nOr install CUDA toolkit first:\n```bash\nconda install cuda -c nvidia\npip install flash-attn --no-build-isolation\n```\n\n**Issue: Slower than expected (no speedup)**\n\nFlash Attention benefits increase with sequence length:\n- \u003C512 tokens: Minimal speedup (10-20%)\n- 512-2K tokens: 2-3x speedup\n- >2K tokens: 3-4x speedup\n\nCheck sequence length is sufficient.\n\n**Issue: RuntimeError: CUDA error**\n\nVerify GPU supports Flash Attention:\n```python\nimport torch\nprint(torch.cuda.get_device_capability())\n# Should be ≥(7, 5) for Turing+\n```\n\nFlash Attention requires:\n- Ampere (A100, A10): ✅ Full support\n- Turing (T4): ✅ Supported\n- Volta (V100): ❌ Not supported\n\n**Issue: Accuracy degradation**\n\nCheck dtype is float16 or bfloat16 (not float32):\n```python\nq = q.to(torch.float16)  # Or torch.bfloat16\n```\n\nFlash Attention uses float16\u002Fbfloat16 for speed. Float32 not supported.\n\n## Advanced topics\n\n**Integration with HuggingFace Transformers**: See [references\u002Ftransformers-integration.md](references\u002Ftransformers-integration.md) for enabling Flash Attention in BERT, GPT, Llama models.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for detailed speed and memory comparisons across GPUs and sequence lengths.\n\n**Algorithm details**: See [references\u002Falgorithm.md](references\u002Falgorithm.md) for tiling strategy, recomputation, and IO complexity analysis.\n\n**Advanced features**: See [references\u002Fadvanced-features.md](references\u002Fadvanced-features.md) for rotary embeddings, ALiBi, paged KV cache, and custom attention masks.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, A10, A30) or AMD MI200+\n- **VRAM**: Same as standard attention (Flash Attention doesn't increase memory)\n- **CUDA**: 12.0+ (11.8 minimum)\n- **PyTorch**: 2.2+ for native support\n\n**Not supported**: V100 (Volta), CPU inference\n\n## Resources\n\n- Paper: \"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\" (NeurIPS 2022)\n- Paper: \"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning\" (ICLR 2024)\n- Blog: https:\u002F\u002Ftridao.me\u002Fblog\u002F2024\u002Fflash3\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention\n- PyTorch docs: https:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Fgenerated\u002Ftorch.nn.functional.scaled_dot_product_attention.html\n\n\n\n",{"license":15,"description":7},"main","35cc2c5cb1f57477268c9f68a3a64363e68de836","agent\u002Fskills\u002Foptimizing-attention-flash","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot","https:\u002F\u002Fgithub.com\u002FSiddhamShah972","2026-08-20 23:29:35",{"source":8,"skillName":5,"commands":31,"prompt":41,"zipAvailable":42},[32,35,38],{"tool":33,"command":34},"npx","npx skills add SiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot --skill optimizing-attention-flash",{"tool":36,"command":37},"bunx","bunx skills add SiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot --skill optimizing-attention-flash",{"tool":39,"command":40},"pnpm","pnpm dlx skills add SiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot --skill optimizing-attention-flash","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Foptimizing-attention-flash\u002FSKILL.md",true,[44,50],{"path":45,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":48,"sourceHtmlUrl":49},"references\u002Fbenchmarks.md","REFERENCE","参考文档",7129,"https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Foptimizing-attention-flash\u002Freferences\u002Fbenchmarks.md",{"path":51,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":52,"sourceHtmlUrl":53},"references\u002Ftransformers-integration.md",7427,"https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Foptimizing-attention-flash\u002Freferences\u002Ftransformers-integration.md",[55,63,70,77,86,94],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":59,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7AZQ1V61JHXWJ370V37AC6D","workorai","WorkorAI talent marketplace skill: candidate job search and employer hiring with white-box match explanations via the WorkorAI MCP server (https:\u002F\u002Fworkorai.com\u002Fmcp). Use when the user asks to find a job, apply to jobs, respond to employer invitations, or when an employer wants to post jobs, search and evaluate candidates, invite them, and review applicants.","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fworkorai\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fworkorai\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7AZP93C6FJ9EXT7QXFQ8HA6","whisper","OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fwhisper\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7AZP4RVG6DEBAS7TPJTRP55","weights-and-biases","Track ML experiments with automatic logging, visualize training in real-time, optimize hyperparameters with sweeps, and manage model registry with W&B - collaborative MLOps platform","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":82,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M7AZNYG21QDFHN1CSGRJ9XJM","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","Apache-2.0","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":90,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M7AZNCEWCKPENGX0ZT274M4Q","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fweb-artifacts-builder\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":19},"01M7AZMQVK156KR60STTBGQGAK","vitest","Set up Vitest testing in any project — detects type (Cloudflare Workers, React, Node, library), generates vitest.config.ts, test setup, utilities, and a sample test. Covers mocking patterns, coverage config, workspace setup, Jest migration. Use whenever the user mentions adding tests, setting up Vitest, configuring tests, migrating from Jest, fixing testing infrastructure, or asks 'how do I test this'.","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fvitest\u002FSKILL.md",[],[]]