[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M737M9RN8MJCYN45G53RQ06B":3,"skill-same:01M737M9RN8MJCYN45G53RQ06B":77},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":49,"repositoryHtmlUrl":50,"repositoryHomepageUrl":6,"ownerHtmlUrl":51,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":52,"install":53,"resources":66},"01M737M9RN8MJCYN45G53RQ06B","flash-attention",null,"Speed up long-sequence transformer training and inference.","ashneil12\u002Fvanilla-hermes-agent-canary","ashneil12","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F49880658?v=4","01M7378BQ9W161EPENEWBYAQNN","01M7378BRDJG8SFS93FG4V6BWY",1,"MIT","https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Foptional-skills\u002Fmlops\u002Fflash-attention\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: flash-attention\ndescription: Speed up long-sequence transformer training and inference.\nversion: 1.0.1\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [flash-attn, torch, transformers]\nplatforms: [linux, macos]\nmetadata:\n  hermes:\n    tags: [Optimization, Flash Attention, Attention Optimization, Memory Efficiency, Speed Optimization, Long Context, PyTorch, SDPA, H100, FP8, Transformers]\n\n---\n\n# Flash Attention - Fast Memory-Efficient Attention\n\n## Quick start\n\nFlash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation.\n\n**PyTorch native (easiest, PyTorch 2.2+)**:\n```python\nimport torch\nimport torch.nn.functional as F\n\nq = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)  # [batch, heads, seq, dim]\nk = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\n\n# Automatically uses Flash Attention if available\nout = F.scaled_dot_product_attention(q, k, v)\n```\n\n**flash-attn library (more features)**:\n```bash\npip install flash-attn --no-build-isolation\n```\n\n```python\nfrom flash_attn import flash_attn_func\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nout = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)\n```\n\n## Common workflows\n\n### Workflow 1: Enable in existing PyTorch model\n\nCopy this checklist:\n\n```\nFlash Attention Integration:\n- [ ] Step 1: Check PyTorch version (≥2.2)\n- [ ] Step 2: Enable Flash Attention backend\n- [ ] Step 3: Verify speedup with profiling\n- [ ] Step 4: Test accuracy matches baseline\n```\n\n**Step 1: Check PyTorch version**\n\n```bash\npython -c \"import torch; print(torch.__version__)\"\n# Should be ≥2.2.0\n```\n\nIf \u003C2.2, upgrade:\n```bash\npip install --upgrade torch\n```\n\n**Step 2: Enable Flash Attention backend**\n\nReplace standard attention:\n```python\n# Before (standard attention)\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F math.sqrt(d_k), dim=-1)\nout = attn_weights @ v\n\n# After (Flash Attention)\nimport torch.nn.functional as F\nout = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)\n```\n\nForce Flash Attention backend (`torch.backends.cuda.sdp_kernel` is deprecated; use\n`torch.nn.attention.sdpa_kernel` with `SDPBackend`):\n```python\nfrom torch.nn.attention import SDPBackend, sdpa_kernel\n\nwith sdpa_kernel(SDPBackend.FLASH_ATTENTION):\n    out = F.scaled_dot_product_attention(q, k, v)\n```\n\n**Step 3: Verify speedup with profiling**\n\n```python\nimport torch.utils.benchmark as benchmark\n\ndef test_attention(use_flash):\n    q, k, v = [torch.randn(2, 8, 2048, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n    if use_flash:\n        from torch.nn.attention import SDPBackend, sdpa_kernel\n        with sdpa_kernel(SDPBackend.FLASH_ATTENTION):\n            return F.scaled_dot_product_attention(q, k, v)\n    else:\n        attn = (q @ k.transpose(-2, -1) \u002F 8.0).softmax(dim=-1)\n        return attn @ v\n\n# Benchmark\nt_flash = benchmark.Timer(stmt='test_attention(True)', globals=globals())\nt_standard = benchmark.Timer(stmt='test_attention(False)', globals=globals())\n\nprint(f\"Flash: {t_flash.timeit(100).mean:.3f}s\")\nprint(f\"Standard: {t_standard.timeit(100).mean:.3f}s\")\n```\n\nExpected: 2-4x speedup for sequences >512 tokens.\n\n**Step 4: Test accuracy matches baseline**\n\n```python\n# Compare outputs\nq, k, v = [torch.randn(1, 8, 512, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Flash Attention\nout_flash = F.scaled_dot_product_attention(q, k, v)\n\n# Standard attention\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F 8.0, dim=-1)\nout_standard = attn_weights @ v\n\n# Check difference\ndiff = (out_flash - out_standard).abs().max()\nprint(f\"Max difference: {diff:.6f}\")\n# Should be \u003C1e-3 for float16\n```\n\n### Workflow 2: Use flash-attn library for advanced features\n\nFor multi-query attention, sliding window, or H100 FP8.\n\nCopy this checklist:\n\n```\nflash-attn Library Setup:\n- [ ] Step 1: Install flash-attn library\n- [ ] Step 2: Modify attention code\n- [ ] Step 3: Enable advanced features\n- [ ] Step 4: Benchmark performance\n```\n\n**Step 1: Install flash-attn library**\n\n```bash\n# NVIDIA GPUs (CUDA 12.0+)\npip install flash-attn --no-build-isolation\n\n# Verify installation\npython -c \"from flash_attn import flash_attn_func; print('Success')\"\n```\n\n**Step 2: Modify attention code**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# Input: [batch_size, seq_len, num_heads, head_dim]\n# Transpose from [batch, heads, seq, dim] if needed\nq = q.transpose(1, 2)  # [batch, seq, heads, dim]\nk = k.transpose(1, 2)\nv = v.transpose(1, 2)\n\nout = flash_attn_func(\n    q, k, v,\n    dropout_p=0.1,\n    causal=True,  # For autoregressive models\n    window_size=(-1, -1),  # No sliding window\n    softmax_scale=None  # Auto-scale\n)\n\nout = out.transpose(1, 2)  # Back to [batch, heads, seq, dim]\n```\n\n**Step 3: Enable advanced features**\n\nMulti-query attention (shared K\u002FV across heads):\n```python\nfrom flash_attn import flash_attn_func\n\n# q: [batch, seq, num_q_heads, dim]\n# k, v: [batch, seq, num_kv_heads, dim]  # Fewer KV heads\nout = flash_attn_func(q, k, v)  # Automatically handles MQA\n```\n\nSliding window attention (local attention):\n```python\n# Only attend to window of 256 tokens before\u002Fafter\nout = flash_attn_func(\n    q, k, v,\n    window_size=(256, 256),  # (left, right) window\n    causal=True\n)\n```\n\n**Step 4: Benchmark performance**\n\n```python\nimport torch\nfrom flash_attn import flash_attn_func\nimport time\n\nq, k, v = [torch.randn(4, 4096, 32, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Warmup\nfor _ in range(10):\n    _ = flash_attn_func(q, k, v)\n\n# Benchmark\ntorch.cuda.synchronize()\nstart = time.time()\nfor _ in range(100):\n    out = flash_attn_func(q, k, v)\n    torch.cuda.synchronize()\nend = time.time()\n\nprint(f\"Time per iteration: {(end-start)\u002F100*1000:.2f}ms\")\nprint(f\"Memory allocated: {torch.cuda.max_memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 3: H100 FP8 optimization (FlashAttention-3)\n\nFor maximum performance on Hopper GPUs (H100).\n\n> **Important:** The pip package `flash-attn` (2.8.x) ships **FlashAttention-2 only** — it does\n> **not** contain FA3 or FP8 H100 kernels, and `flash_attn_func` does **not** auto-use FP8.\n> FlashAttention-3 is a separate **beta** build compiled from source from the repo's `hopper\u002F`\n> directory, exposed via the `flash_attn_interface` module. FA3 supports FP16\u002FBF16 forward+backward\n> and **FP8 forward only**.\n\n```\nFP8 Setup:\n- [ ] Step 1: Verify Hopper (H100) GPU available\n- [ ] Step 2: Build & install FlashAttention-3 from source (hopper\u002F)\n- [ ] Step 3: Use the FA3 interface (FP8 forward)\n```\n\n**Step 1: Verify H100 GPU**\n\n```bash\nnvidia-smi --query-gpu=name --format=csv\n# Should show \"H100\" or \"H800\"\n```\n\n**Step 2: Build & install FlashAttention-3 from source**\n\nFA3 is NOT included in `pip install flash-attn`. Build it from the `hopper\u002F` subdirectory:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention.git\ncd flash-attention\u002Fhopper\npython setup.py install\n# (compilation is heavy and requires a CUDA toolchain + Hopper GPU)\n```\n\n**Step 3: Use the FA3 interface (FP8 forward)**\n\nFA3 exposes its own module `flash_attn_interface` (distinct from the FA2 `flash_attn`).\nFP8 is a **forward-only** path and expects `float8_e4m3fn` inputs:\n\n```python\nimport torch\nfrom flash_attn_interface import flash_attn_func  # FA3 (hopper build), not `flash_attn`\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nq = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nk = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\n\n# FP8 forward (inference \u002F forward-only): cast to float8_e4m3fn\nq_fp8 = q.to(torch.float8_e4m3fn)\nk_fp8 = k.to(torch.float8_e4m3fn)\nv_fp8 = v.to(torch.float8_e4m3fn)\n\nout = flash_attn_func(q_fp8, k_fp8, v_fp8, causal=True)\n# FP16\u002FBF16 forward+backward is also supported by the FA3 interface.\n```\n\n## When to use vs alternatives\n\n**Use Flash Attention when:**\n- Training transformers with sequences >512 tokens\n- Running inference with long context (>2K tokens)\n- GPU memory constrained (OOM with standard attention)\n- Need 2-4x speedup without accuracy loss\n- Using PyTorch 2.2+ or can install flash-attn\n\n**Use alternatives instead:**\n- **Standard attention**: Sequences \u003C256 tokens (overhead not worth it)\n- **xFormers**: Need more attention variants (not just speed)\n- **Memory-efficient attention**: CPU inference (Flash Attention needs GPU)\n\n## Common issues\n\n**Issue: ImportError: cannot import flash_attn**\n\nInstall with no-build-isolation flag:\n```bash\npip install flash-attn --no-build-isolation\n```\n\nOr install CUDA toolkit first:\n```bash\nconda install cuda -c nvidia\npip install flash-attn --no-build-isolation\n```\n\n**Issue: Slower than expected (no speedup)**\n\nFlash Attention benefits increase with sequence length:\n- \u003C512 tokens: Minimal speedup (10-20%)\n- 512-2K tokens: 2-3x speedup\n- >2K tokens: 3-4x speedup\n\nCheck sequence length is sufficient.\n\n**Issue: RuntimeError: CUDA error**\n\nVerify GPU supports Flash Attention:\n```python\nimport torch\nprint(torch.cuda.get_device_capability())\n# Should be ≥(7, 5) for Turing+\n```\n\nFlash Attention requires:\n- Ampere (A100, A10): ✅ Full support\n- Turing (T4): ✅ Supported\n- Volta (V100): ❌ Not supported\n\n**Issue: Accuracy degradation**\n\nCheck dtype is float16 or bfloat16 (not float32):\n```python\nq = q.to(torch.float16)  # Or torch.bfloat16\n```\n\nFlash Attention uses float16\u002Fbfloat16 for speed. Float32 not supported.\n\n## Advanced topics\n\n**Integration with HuggingFace Transformers**: See [references\u002Ftransformers-integration.md](references\u002Ftransformers-integration.md) for enabling Flash Attention in BERT, GPT, Llama models.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for detailed speed and memory comparisons across GPUs and sequence lengths.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, A10, A30) or AMD MI200+\n- **VRAM**: Same as standard attention (Flash Attention doesn't increase memory)\n- **CUDA**: 12.0+ (11.8 minimum)\n- **PyTorch**: 2.2+ for native support\n\n**Not supported**: V100 (Volta), CPU inference\n\n## Resources\n\n- Paper: \"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\" (NeurIPS 2022)\n- Paper: \"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning\" (ICLR 2024)\n- Blog: https:\u002F\u002Ftridao.me\u002Fblog\u002F2024\u002Fflash3\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention\n- PyTorch docs: https:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Fgenerated\u002Ftorch.nn.functional.scaled_dot_product_attention.html\n\n\n\n","# Flash Attention - Fast Memory-Efficient Attention\n\n## Quick start\n\nFlash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation.\n\n**PyTorch native (easiest, PyTorch 2.2+)**:\n```python\nimport torch\nimport torch.nn.functional as F\n\nq = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)  # [batch, heads, seq, dim]\nk = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\n\n# Automatically uses Flash Attention if available\nout = F.scaled_dot_product_attention(q, k, v)\n```\n\n**flash-attn library (more features)**:\n```bash\npip install flash-attn --no-build-isolation\n```\n\n```python\nfrom flash_attn import flash_attn_func\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nout = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)\n```\n\n## Common workflows\n\n### Workflow 1: Enable in existing PyTorch model\n\nCopy this checklist:\n\n```\nFlash Attention Integration:\n- [ ] Step 1: Check PyTorch version (≥2.2)\n- [ ] Step 2: Enable Flash Attention backend\n- [ ] Step 3: Verify speedup with profiling\n- [ ] Step 4: Test accuracy matches baseline\n```\n\n**Step 1: Check PyTorch version**\n\n```bash\npython -c \"import torch; print(torch.__version__)\"\n# Should be ≥2.2.0\n```\n\nIf \u003C2.2, upgrade:\n```bash\npip install --upgrade torch\n```\n\n**Step 2: Enable Flash Attention backend**\n\nReplace standard attention:\n```python\n# Before (standard attention)\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F math.sqrt(d_k), dim=-1)\nout = attn_weights @ v\n\n# After (Flash Attention)\nimport torch.nn.functional as F\nout = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)\n```\n\nForce Flash Attention backend (`torch.backends.cuda.sdp_kernel` is deprecated; use\n`torch.nn.attention.sdpa_kernel` with `SDPBackend`):\n```python\nfrom torch.nn.attention import SDPBackend, sdpa_kernel\n\nwith sdpa_kernel(SDPBackend.FLASH_ATTENTION):\n    out = F.scaled_dot_product_attention(q, k, v)\n```\n\n**Step 3: Verify speedup with profiling**\n\n```python\nimport torch.utils.benchmark as benchmark\n\ndef test_attention(use_flash):\n    q, k, v = [torch.randn(2, 8, 2048, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n    if use_flash:\n        from torch.nn.attention import SDPBackend, sdpa_kernel\n        with sdpa_kernel(SDPBackend.FLASH_ATTENTION):\n            return F.scaled_dot_product_attention(q, k, v)\n    else:\n        attn = (q @ k.transpose(-2, -1) \u002F 8.0).softmax(dim=-1)\n        return attn @ v\n\n# Benchmark\nt_flash = benchmark.Timer(stmt='test_attention(True)', globals=globals())\nt_standard = benchmark.Timer(stmt='test_attention(False)', globals=globals())\n\nprint(f\"Flash: {t_flash.timeit(100).mean:.3f}s\")\nprint(f\"Standard: {t_standard.timeit(100).mean:.3f}s\")\n```\n\nExpected: 2-4x speedup for sequences >512 tokens.\n\n**Step 4: Test accuracy matches baseline**\n\n```python\n# Compare outputs\nq, k, v = [torch.randn(1, 8, 512, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Flash Attention\nout_flash = F.scaled_dot_product_attention(q, k, v)\n\n# Standard attention\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F 8.0, dim=-1)\nout_standard = attn_weights @ v\n\n# Check difference\ndiff = (out_flash - out_standard).abs().max()\nprint(f\"Max difference: {diff:.6f}\")\n# Should be \u003C1e-3 for float16\n```\n\n### Workflow 2: Use flash-attn library for advanced features\n\nFor multi-query attention, sliding window, or H100 FP8.\n\nCopy this checklist:\n\n```\nflash-attn Library Setup:\n- [ ] Step 1: Install flash-attn library\n- [ ] Step 2: Modify attention code\n- [ ] Step 3: Enable advanced features\n- [ ] Step 4: Benchmark performance\n```\n\n**Step 1: Install flash-attn library**\n\n```bash\n# NVIDIA GPUs (CUDA 12.0+)\npip install flash-attn --no-build-isolation\n\n# Verify installation\npython -c \"from flash_attn import flash_attn_func; print('Success')\"\n```\n\n**Step 2: Modify attention code**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# Input: [batch_size, seq_len, num_heads, head_dim]\n# Transpose from [batch, heads, seq, dim] if needed\nq = q.transpose(1, 2)  # [batch, seq, heads, dim]\nk = k.transpose(1, 2)\nv = v.transpose(1, 2)\n\nout = flash_attn_func(\n    q, k, v,\n    dropout_p=0.1,\n    causal=True,  # For autoregressive models\n    window_size=(-1, -1),  # No sliding window\n    softmax_scale=None  # Auto-scale\n)\n\nout = out.transpose(1, 2)  # Back to [batch, heads, seq, dim]\n```\n\n**Step 3: Enable advanced features**\n\nMulti-query attention (shared K\u002FV across heads):\n```python\nfrom flash_attn import flash_attn_func\n\n# q: [batch, seq, num_q_heads, dim]\n# k, v: [batch, seq, num_kv_heads, dim]  # Fewer KV heads\nout = flash_attn_func(q, k, v)  # Automatically handles MQA\n```\n\nSliding window attention (local attention):\n```python\n# Only attend to window of 256 tokens before\u002Fafter\nout = flash_attn_func(\n    q, k, v,\n    window_size=(256, 256),  # (left, right) window\n    causal=True\n)\n```\n\n**Step 4: Benchmark performance**\n\n```python\nimport torch\nfrom flash_attn import flash_attn_func\nimport time\n\nq, k, v = [torch.randn(4, 4096, 32, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Warmup\nfor _ in range(10):\n    _ = flash_attn_func(q, k, v)\n\n# Benchmark\ntorch.cuda.synchronize()\nstart = time.time()\nfor _ in range(100):\n    out = flash_attn_func(q, k, v)\n    torch.cuda.synchronize()\nend = time.time()\n\nprint(f\"Time per iteration: {(end-start)\u002F100*1000:.2f}ms\")\nprint(f\"Memory allocated: {torch.cuda.max_memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 3: H100 FP8 optimization (FlashAttention-3)\n\nFor maximum performance on Hopper GPUs (H100).\n\n> **Important:** The pip package `flash-attn` (2.8.x) ships **FlashAttention-2 only** — it does\n> **not** contain FA3 or FP8 H100 kernels, and `flash_attn_func` does **not** auto-use FP8.\n> FlashAttention-3 is a separate **beta** build compiled from source from the repo's `hopper\u002F`\n> directory, exposed via the `flash_attn_interface` module. FA3 supports FP16\u002FBF16 forward+backward\n> and **FP8 forward only**.\n\n```\nFP8 Setup:\n- [ ] Step 1: Verify Hopper (H100) GPU available\n- [ ] Step 2: Build & install FlashAttention-3 from source (hopper\u002F)\n- [ ] Step 3: Use the FA3 interface (FP8 forward)\n```\n\n**Step 1: Verify H100 GPU**\n\n```bash\nnvidia-smi --query-gpu=name --format=csv\n# Should show \"H100\" or \"H800\"\n```\n\n**Step 2: Build & install FlashAttention-3 from source**\n\nFA3 is NOT included in `pip install flash-attn`. Build it from the `hopper\u002F` subdirectory:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention.git\ncd flash-attention\u002Fhopper\npython setup.py install\n# (compilation is heavy and requires a CUDA toolchain + Hopper GPU)\n```\n\n**Step 3: Use the FA3 interface (FP8 forward)**\n\nFA3 exposes its own module `flash_attn_interface` (distinct from the FA2 `flash_attn`).\nFP8 is a **forward-only** path and expects `float8_e4m3fn` inputs:\n\n```python\nimport torch\nfrom flash_attn_interface import flash_attn_func  # FA3 (hopper build), not `flash_attn`\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nq = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nk = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\n\n# FP8 forward (inference \u002F forward-only): cast to float8_e4m3fn\nq_fp8 = q.to(torch.float8_e4m3fn)\nk_fp8 = k.to(torch.float8_e4m3fn)\nv_fp8 = v.to(torch.float8_e4m3fn)\n\nout = flash_attn_func(q_fp8, k_fp8, v_fp8, causal=True)\n# FP16\u002FBF16 forward+backward is also supported by the FA3 interface.\n```\n\n## When to use vs alternatives\n\n**Use Flash Attention when:**\n- Training transformers with sequences >512 tokens\n- Running inference with long context (>2K tokens)\n- GPU memory constrained (OOM with standard attention)\n- Need 2-4x speedup without accuracy loss\n- Using PyTorch 2.2+ or can install flash-attn\n\n**Use alternatives instead:**\n- **Standard attention**: Sequences \u003C256 tokens (overhead not worth it)\n- **xFormers**: Need more attention variants (not just speed)\n- **Memory-efficient attention**: CPU inference (Flash Attention needs GPU)\n\n## Common issues\n\n**Issue: ImportError: cannot import flash_attn**\n\nInstall with no-build-isolation flag:\n```bash\npip install flash-attn --no-build-isolation\n```\n\nOr install CUDA toolkit first:\n```bash\nconda install cuda -c nvidia\npip install flash-attn --no-build-isolation\n```\n\n**Issue: Slower than expected (no speedup)**\n\nFlash Attention benefits increase with sequence length:\n- \u003C512 tokens: Minimal speedup (10-20%)\n- 512-2K tokens: 2-3x speedup\n- >2K tokens: 3-4x speedup\n\nCheck sequence length is sufficient.\n\n**Issue: RuntimeError: CUDA error**\n\nVerify GPU supports Flash Attention:\n```python\nimport torch\nprint(torch.cuda.get_device_capability())\n# Should be ≥(7, 5) for Turing+\n```\n\nFlash Attention requires:\n- Ampere (A100, A10): ✅ Full support\n- Turing (T4): ✅ Supported\n- Volta (V100): ❌ Not supported\n\n**Issue: Accuracy degradation**\n\nCheck dtype is float16 or bfloat16 (not float32):\n```python\nq = q.to(torch.float16)  # Or torch.bfloat16\n```\n\nFlash Attention uses float16\u002Fbfloat16 for speed. Float32 not supported.\n\n## Advanced topics\n\n**Integration with HuggingFace Transformers**: See [references\u002Ftransformers-integration.md](references\u002Ftransformers-integration.md) for enabling Flash Attention in BERT, GPT, Llama models.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for detailed speed and memory comparisons across GPUs and sequence lengths.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, A10, A30) or AMD MI200+\n- **VRAM**: Same as standard attention (Flash Attention doesn't increase memory)\n- **CUDA**: 12.0+ (11.8 minimum)\n- **PyTorch**: 2.2+ for native support\n\n**Not supported**: V100 (Volta), CPU inference\n\n## Resources\n\n- Paper: \"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\" (NeurIPS 2022)\n- Paper: \"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning\" (ICLR 2024)\n- Blog: https:\u002F\u002Ftridao.me\u002Fblog\u002F2024\u002Fflash3\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention\n- PyTorch docs: https:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Fgenerated\u002Ftorch.nn.functional.scaled_dot_product_attention.html\n\n\n\n",{"name":5,"author":23,"license":14,"version":24,"metadata":25,"platforms":39,"description":7,"dependencies":42},"Orchestra Research","1.0.1",{"hermes":26},{"tags":27},[28,29,30,31,32,33,34,35,36,37,38],"Optimization","Flash Attention","Attention Optimization","Memory Efficiency","Speed Optimization","Long Context","PyTorch","SDPA","H100","FP8","Transformers",[40,41],"linux","macos",[43,44,45],"flash-attn","torch","transformers","main","0905b0264ce5a54908bedc9637e50251543db132","optional-skills\u002Fmlops\u002Fflash-attention",0,"https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary","https:\u002F\u002Fgithub.com\u002Fashneil12","2026-11-04 08:25:00",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add ashneil12\u002Fvanilla-hermes-agent-canary --skill flash-attention",{"tool":59,"command":60},"bunx","bunx skills add ashneil12\u002Fvanilla-hermes-agent-canary --skill flash-attention",{"tool":62,"command":63},"pnpm","pnpm dlx skills add ashneil12\u002Fvanilla-hermes-agent-canary --skill flash-attention","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Foptional-skills\u002Fmlops\u002Fflash-attention\u002FSKILL.md",true,[67,73],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"references\u002Fbenchmarks.md","REFERENCE","参考文档",7129,"https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Foptional-skills\u002Fmlops\u002Fflash-attention\u002Freferences\u002Fbenchmarks.md",{"path":74,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Ftransformers-integration.md",7427,"https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Foptional-skills\u002Fmlops\u002Fflash-attention\u002Freferences\u002Ftransformers-integration.md",[78,85,92,99,106,113],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":18},"01M7380YR13A23353XT9NS6EGR","blocked-page-recovery","Use when a fetch fails: 403\u002F429, paywall, WAF, bot wall.","https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Fskills\u002Fweb\u002Fblocked-page-recovery\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":18},"01M7380VZ8P60Q64J3F92D4AMY","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":18},"01M7380QXN6F90R0MH1M3FHRKX","systematic-debugging","4-phase root cause debugging: understand bugs before fixing.","https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":18},"01M7380N4BZYT2WRRZN1N9DZ65","spike","Throwaway experiments to validate an idea before build.","https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Fskills\u002Fsoftware-development\u002Fspike\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":18},"01M7380JAEC9ZH5DVPHHK56P97","simplify-code","Parallel 4-agent cleanup of recent code changes.","https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Fskills\u002Fsoftware-development\u002Fsimplify-code\u002FSKILL.md",[],[],{"skillNo":114,"name":115,"nameZh":6,"description":116,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":117,"categories":118,"keywords":119,"publishedAt":18},"01M7380F5MBB7FKWWV3G3TZ193","requesting-code-review","Pre-commit review: security scan, quality gates, auto-fix.","https:\u002F\u002Fgithub.com\u002Fashneil12\u002Fvanilla-hermes-agent-canary\u002Fblob\u002Fmain\u002Fskills\u002Fsoftware-development\u002Frequesting-code-review\u002FSKILL.md",[],[]]