[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EWPR2YPM99XWP6N2FP6WKP":3,"skill-same:01M6EWPR2YPM99XWP6N2FP6WKP":96},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":60,"commitSha":61,"skillDirectory":62,"forksCount":14,"repositoryHtmlUrl":63,"repositoryHomepageUrl":64,"ownerHtmlUrl":65,"ownerBlogUrl":6,"ownerDescription":66,"updatedAt":67,"install":68,"resources":81},"01M6EWPR2YPM99XWP6N2FP6WKP","long-context",null,"Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.","henriquescastilho\u002Fmy-claude","henriquescastilho","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F153296945?v=4","01M6ESH7EXPXTH9CVQ4K16SRG8","Henrique Scheer","01M6ESH7JHEK2PTNNH4HEH787Z",0,"MIT","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-long-context\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"long","long context","context","strategies","Transformer","RoPE","Emerging Techniques","processing","tokens","bias","attention","llms","methods","use","flash","Transformers","embeddings","yarn","documents","windows","2026-11-10 10:12:30",false,"---\nname: long-context\ndescription: Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, Long Context, RoPE, YaRN, ALiBi, Position Interpolation, Extended Context, Rotary Embeddings, Attention Bias, Context Extension, Positional Encoding]\ndependencies: [transformers, torch, flash-attn]\n---\n\n# Long Context: Extending Transformer Context Windows\n\n## When to Use This Skill\n\nUse Long Context techniques when you need to:\n- **Process long documents** (32k, 64k, 128k+ tokens) with transformer models\n- **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.)\n- **Implement efficient positional encodings** (RoPE, ALiBi)\n- **Train models** with length extrapolation capabilities\n- **Deploy models** that handle variable-length inputs efficiently\n- **Fine-tune** existing models for longer contexts with minimal compute\n\n**Key Techniques**: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation\n\n**Papers**: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595)\n\n## Installation\n\n```bash\n# HuggingFace Transformers (includes RoPE, YaRN support)\npip install transformers torch\n\n# For custom implementations\npip install einops  # Tensor operations\npip install rotary-embedding-torch  # Standalone RoPE\n\n# Optional: FlashAttention for efficiency\npip install flash-attn --no-build-isolation\n```\n\n## Quick Start\n\n### RoPE (Rotary Position Embeddings)\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass RotaryEmbedding(nn.Module):\n    \"\"\"Rotary Position Embeddings (RoPE).\"\"\"\n\n    def __init__(self, dim, max_seq_len=8192, base=10000):\n        super().__init__()\n        # Compute inverse frequencies\n        inv_freq = 1.0 \u002F (base ** (torch.arange(0, dim, 2).float() \u002F dim))\n        self.register_buffer(\"inv_freq\", inv_freq)\n        self.max_seq_len = max_seq_len\n\n    def forward(self, seq_len, device):\n        # Position indices\n        t = torch.arange(seq_len, device=device).type_as(self.inv_freq)\n\n        # Compute frequencies\n        freqs = torch.outer(t, self.inv_freq)  # (seq_len, dim\u002F2)\n\n        # Compute sin and cos\n        emb = torch.cat((freqs, freqs), dim=-1)  # (seq_len, dim)\n        return emb.cos(), emb.sin()\n\ndef rotate_half(x):\n    \"\"\"Rotate half the hidden dimensions.\"\"\"\n    x1, x2 = x.chunk(2, dim=-1)\n    return torch.cat((-x2, x1), dim=-1)\n\ndef apply_rotary_pos_emb(q, k, cos, sin):\n    \"\"\"Apply rotary embeddings to queries and keys.\"\"\"\n    # q, k shape: (batch, heads, seq_len, dim)\n    q_embed = (q * cos) + (rotate_half(q) * sin)\n    k_embed = (k * cos) + (rotate_half(k) * sin)\n    return q_embed, k_embed\n\n# Usage\nrope = RotaryEmbedding(dim=64, max_seq_len=8192)\ncos, sin = rope(seq_len=2048, device='cuda')\n\n# In attention layer\nq_rotated, k_rotated = apply_rotary_pos_emb(query, key, cos, sin)\n```\n\n### ALiBi (Attention with Linear Biases)\n\n```python\ndef get_alibi_slopes(num_heads):\n    \"\"\"Get ALiBi slope values for each attention head.\"\"\"\n    def get_slopes_power_of_2(n):\n        start = 2 ** (-(2 ** -(math.log2(n) - 3)))\n        ratio = start\n        return [start * (ratio ** i) for i in range(n)]\n\n    if math.log2(num_heads).is_integer():\n        return get_slopes_power_of_2(num_heads)\n    else:\n        # Closest power of 2\n        closest_power = 2 ** math.floor(math.log2(num_heads))\n        slopes = get_slopes_power_of_2(closest_power)\n        # Add extra slopes\n        extra = get_slopes_power_of_2(2 * closest_power)\n        slopes.extend(extra[0::2][:num_heads - closest_power])\n        return slopes\n\ndef create_alibi_bias(seq_len, num_heads):\n    \"\"\"Create ALiBi attention bias.\"\"\"\n    # Distance matrix\n    context_position = torch.arange(seq_len)\n    memory_position = torch.arange(seq_len)\n    relative_position = memory_position[None, :] - context_position[:, None]\n\n    # Get slopes\n    slopes = torch.tensor(get_alibi_slopes(num_heads))\n\n    # Apply slopes to distances\n    alibi = slopes[:, None, None] * relative_position[None, :, :]\n    return alibi  # (num_heads, seq_len, seq_len)\n\n# Usage in attention\nnum_heads = 8\nseq_len = 2048\nalibi_bias = create_alibi_bias(seq_len, num_heads).to('cuda')\n\n# Add bias to attention scores\n# attn_scores shape: (batch, num_heads, seq_len, seq_len)\nattn_scores = attn_scores + alibi_bias\nattn_weights = torch.softmax(attn_scores, dim=-1)\n```\n\n### Position Interpolation for LLaMA\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\n# Original context: 2048 tokens\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Extend to 32k with position interpolation\n# Modify RoPE base frequency\nmodel.config.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 16.0  # 2048 * 16 = 32768\n}\n\n# Or use dynamic scaling\nmodel.config.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 16.0\n}\n\n# Fine-tune with long documents (minimal steps needed)\n# Position interpolation works out-of-the-box after this config change\n```\n\n## Core Concepts\n\n### 1. RoPE (Rotary Position Embeddings)\n\n**How it works:**\n- Encodes absolute position via rotation matrix\n- Provides relative position dependency in attention\n- Enables length extrapolation\n\n**Mathematical formulation:**\n```\nq_m = (W_q * x_m) * e^(imθ)\nk_n = (W_k * x_n) * e^(inθ)\n\nwhere θ_j = base^(-2j\u002Fd) for j ∈ [0, d\u002F2)\n```\n\n**Advantages:**\n- Decaying inter-token dependency with distance\n- Compatible with linear attention\n- Better extrapolation than absolute position encodings\n\n### 2. YaRN (Yet another RoPE extensioN)\n\n**Key innovation:**\n- NTK-aware interpolation (Neural Tangent Kernel)\n- Attention temperature scaling\n- Efficient context extension (10× less tokens vs baselines)\n\n**Parameters:**\n```python\n# YaRN configuration\nyarn_config = {\n    \"scale\": 16,                    # Extension factor\n    \"original_max_position\": 2048,  # Base context\n    \"extrapolation_factor\": 1.0,    # NTK parameter\n    \"attn_factor\": 1.0,             # Attention scaling\n    \"beta_fast\": 32,                # High-frequency scale\n    \"beta_slow\": 1,                 # Low-frequency scale\n}\n```\n\n**Performance:**\n- Extends LLaMA to 128k tokens\n- 2.5× less training steps than baselines\n- State-of-the-art context window extension\n\n### 3. ALiBi (Attention with Linear Biases)\n\n**Core idea:**\n- No positional embeddings added to tokens\n- Apply distance penalty directly to attention scores\n- Bias proportional to key-query distance\n\n**Formula:**\n```\nattention_bias[i, j] = -m * |i - j|\n\nwhere m = slope for each attention head\n```\n\n**Advantages:**\n- 11% faster training vs sinusoidal embeddings\n- 11% less memory usage\n- Strong length extrapolation (train 1k, test 2k+)\n- Inductive bias towards recency\n\n### 4. Position Interpolation\n\n**Technique:**\n- Linearly down-scale position indices\n- Interpolate within trained range (vs extrapolate beyond)\n- Minimal fine-tuning required\n\n**Formula:**\n```\n# Original: position indices [0, 1, 2, ..., L]\n# Extended: position indices [0, 0.5, 1.0, ..., L\u002F2]\n# (for 2× extension)\n\nscaled_position[i] = i \u002F extension_factor\n```\n\n**Results:**\n- LLaMA 7B-65B extended to 32k tokens\n- 1000 fine-tuning steps sufficient\n- 600× better stability than extrapolation\n\n## Method Comparison\n\n| Method | Max Context | Training Needed | Memory | Extrapolation | Best For |\n|--------|-------------|-----------------|--------|---------------|----------|\n| **RoPE** | 8k-32k | Full pre-training | Moderate | Good | New models |\n| **YaRN** | 32k-128k | Minimal (10× efficient) | Moderate | Excellent | Extending existing models |\n| **ALiBi** | Unlimited | Full pre-training | Low (-11%) | Excellent | Training from scratch |\n| **Position Interpolation** | 32k+ | Minimal (1k steps) | Moderate | Poor (by design) | Quick extension |\n\n## Implementation Patterns\n\n### HuggingFace Transformers Integration\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoConfig\n\n# RoPE with YaRN scaling\nconfig = AutoConfig.from_pretrained(\"mistralai\u002FMistral-7B-v0.1\")\nconfig.rope_scaling = {\n    \"type\": \"yarn\",\n    \"factor\": 8.0,\n    \"original_max_position_embeddings\": 8192,\n    \"attention_factor\": 1.0\n}\n\nmodel = AutoModelForCausalLM.from_config(config)\n\n# Position interpolation (simpler)\nconfig.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 4.0\n}\n\n# Dynamic scaling (adjusts based on input length)\nconfig.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 8.0\n}\n```\n\n### Custom RoPE Implementation\n\n```python\nclass LongContextAttention(nn.Module):\n    \"\"\"Multi-head attention with RoPE.\"\"\"\n\n    def __init__(self, hidden_size, num_heads, max_seq_len=32768):\n        super().__init__()\n        self.num_heads = num_heads\n        self.head_dim = hidden_size \u002F\u002F num_heads\n\n        # Q, K, V projections\n        self.q_proj = nn.Linear(hidden_size, hidden_size)\n        self.k_proj = nn.Linear(hidden_size, hidden_size)\n        self.v_proj = nn.Linear(hidden_size, hidden_size)\n        self.o_proj = nn.Linear(hidden_size, hidden_size)\n\n        # RoPE\n        self.rotary_emb = RotaryEmbedding(\n            dim=self.head_dim,\n            max_seq_len=max_seq_len\n        )\n\n    def forward(self, hidden_states):\n        batch_size, seq_len, _ = hidden_states.shape\n\n        # Project to Q, K, V\n        q = self.q_proj(hidden_states)\n        k = self.k_proj(hidden_states)\n        v = self.v_proj(hidden_states)\n\n        # Reshape for multi-head\n        q = q.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        k = k.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        v = v.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n\n        # Apply RoPE\n        cos, sin = self.rotary_emb(seq_len, device=hidden_states.device)\n        q, k = apply_rotary_pos_emb(q, k, cos, sin)\n\n        # Standard attention\n        attn_output = F.scaled_dot_product_attention(q, k, v)\n\n        # Reshape and project\n        attn_output = attn_output.transpose(1, 2).contiguous()\n        attn_output = attn_output.view(batch_size, seq_len, -1)\n        output = self.o_proj(attn_output)\n\n        return output\n```\n\n## Fine-tuning for Long Context\n\n### Minimal Fine-tuning (Position Interpolation)\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\n# Extend model config\nmodel.config.max_position_embeddings = 32768\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n\n# Training args (minimal steps needed)\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fllama-32k\",\n    num_train_epochs=1,\n    max_steps=1000,           # Only 1000 steps!\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n    learning_rate=2e-5,\n    warmup_steps=100,\n    logging_steps=10,\n    save_steps=500,\n)\n\n# Train on long documents\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=long_document_dataset,  # 32k token sequences\n)\n\ntrainer.train()\n```\n\n### YaRN Fine-tuning\n\n```bash\n# Clone YaRN implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\ncd yarn\n\n# Fine-tune LLaMA with YaRN\npython scripts\u002Ftrain.py \\\n    --model meta-llama\u002FLlama-2-7b-hf \\\n    --scale 16 \\\n    --rope_theta 10000 \\\n    --max_length 32768 \\\n    --batch_size 1 \\\n    --gradient_accumulation 16 \\\n    --steps 400 \\\n    --learning_rate 2e-5\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# For NEW models (training from scratch)\nuse_method = \"ALiBi\"  # Best extrapolation, lowest memory\n\n# For EXTENDING existing RoPE models\nuse_method = \"YaRN\"  # Most efficient extension (10× less data)\n\n# For QUICK extension with minimal compute\nuse_method = \"Position Interpolation\"  # 1000 steps\n\n# For MODERATE extension with good efficiency\nuse_method = \"Linear RoPE Scaling\"  # Built-in, simple\n```\n\n### 2. Scaling Factor Selection\n\n```python\n# Conservative (safer, better quality)\nscaling_factor = 2.0  # 8k → 16k\n\n# Moderate (good balance)\nscaling_factor = 4.0  # 8k → 32k\n\n# Aggressive (requires more fine-tuning)\nscaling_factor = 8.0  # 8k → 64k\nscaling_factor = 16.0  # 8k → 128k\n\n# Rule: Larger factors need more fine-tuning steps\nsteps_needed = 100 * scaling_factor  # Rough estimate\n```\n\n### 3. Fine-tuning Data\n\n```python\n# ✅ Good: Long documents matching target length\ntrain_data = [\n    {\"text\": long_doc_32k_tokens},  # Full 32k\n    {\"text\": long_doc_24k_tokens},  # Varied lengths\n    {\"text\": long_doc_16k_tokens},\n]\n\n# ❌ Bad: Short documents (won't learn long context)\ntrain_data = [\n    {\"text\": short_doc_2k_tokens},\n]\n\n# Use datasets like:\n# - PG-19 (books, long texts)\n# - arXiv papers\n# - Long-form conversations\n# - GitHub repositories (concatenated files)\n```\n\n### 4. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Applying position interpolation without fine-tuning\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n# Model will perform poorly without fine-tuning!\n\n# ✅ Good: Fine-tune after scaling\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\nfine_tune(model, long_documents, steps=1000)\n\n# ❌ Bad: Too aggressive scaling without data\nscale_to_1M_tokens()  # Won't work without massive fine-tuning\n\n# ✅ Good: Incremental scaling\n# 8k → 16k → 32k → 64k (fine-tune at each step)\n```\n\n## Production Deployment\n\n### Inference with Long Context\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load long-context model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"togethercomputer\u002FLLaMA-2-7B-32K\",  # 32k context\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"togethercomputer\u002FLLaMA-2-7B-32K\")\n\n# Process long document\nlong_text = \"...\" * 30000  # 30k tokens\ninputs = tokenizer(long_text, return_tensors=\"pt\", truncation=False).to('cuda')\n\n# Generate\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=512,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Memory Optimization\n\n```python\n# Use gradient checkpointing for fine-tuning\nmodel.gradient_checkpointing_enable()\n\n# Use Flash Attention 2\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    attn_implementation=\"flash_attention_2\",  # 2-3× faster\n    torch_dtype=torch.float16\n)\n\n# Use paged attention (vLLM)\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"togethercomputer\u002FLLaMA-2-7B-32K\",\n    max_model_len=32768,  # 32k context\n    gpu_memory_utilization=0.9\n)\n```\n\n## Resources\n\n- **RoPE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.09864 (RoFormer)\n- **YaRN Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.00071\n- **ALiBi Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2108.12409 (Train Short, Test Long)\n- **Position Interpolation**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.15595\n- **HuggingFace RoPE Utils**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fmain\u002Fsrc\u002Ftransformers\u002Fmodeling_rope_utils.py\n- **YaRN Implementation**: https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\n- **Together AI Blog**: https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fllama-2-7b-32k\n\n## See Also\n\n- `references\u002Frope.md` - Detailed RoPE implementation and theory\n- `references\u002Fextension_methods.md` - YaRN, ALiBi, Position Interpolation comparisons\n- `references\u002Ffine_tuning.md` - Complete fine-tuning guide for context extension\n\n\n","# Long Context: Extending Transformer Context Windows\n\n## When to Use This Skill\n\nUse Long Context techniques when you need to:\n- **Process long documents** (32k, 64k, 128k+ tokens) with transformer models\n- **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.)\n- **Implement efficient positional encodings** (RoPE, ALiBi)\n- **Train models** with length extrapolation capabilities\n- **Deploy models** that handle variable-length inputs efficiently\n- **Fine-tune** existing models for longer contexts with minimal compute\n\n**Key Techniques**: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation\n\n**Papers**: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595)\n\n## Installation\n\n```bash\n# HuggingFace Transformers (includes RoPE, YaRN support)\npip install transformers torch\n\n# For custom implementations\npip install einops  # Tensor operations\npip install rotary-embedding-torch  # Standalone RoPE\n\n# Optional: FlashAttention for efficiency\npip install flash-attn --no-build-isolation\n```\n\n## Quick Start\n\n### RoPE (Rotary Position Embeddings)\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass RotaryEmbedding(nn.Module):\n    \"\"\"Rotary Position Embeddings (RoPE).\"\"\"\n\n    def __init__(self, dim, max_seq_len=8192, base=10000):\n        super().__init__()\n        # Compute inverse frequencies\n        inv_freq = 1.0 \u002F (base ** (torch.arange(0, dim, 2).float() \u002F dim))\n        self.register_buffer(\"inv_freq\", inv_freq)\n        self.max_seq_len = max_seq_len\n\n    def forward(self, seq_len, device):\n        # Position indices\n        t = torch.arange(seq_len, device=device).type_as(self.inv_freq)\n\n        # Compute frequencies\n        freqs = torch.outer(t, self.inv_freq)  # (seq_len, dim\u002F2)\n\n        # Compute sin and cos\n        emb = torch.cat((freqs, freqs), dim=-1)  # (seq_len, dim)\n        return emb.cos(), emb.sin()\n\ndef rotate_half(x):\n    \"\"\"Rotate half the hidden dimensions.\"\"\"\n    x1, x2 = x.chunk(2, dim=-1)\n    return torch.cat((-x2, x1), dim=-1)\n\ndef apply_rotary_pos_emb(q, k, cos, sin):\n    \"\"\"Apply rotary embeddings to queries and keys.\"\"\"\n    # q, k shape: (batch, heads, seq_len, dim)\n    q_embed = (q * cos) + (rotate_half(q) * sin)\n    k_embed = (k * cos) + (rotate_half(k) * sin)\n    return q_embed, k_embed\n\n# Usage\nrope = RotaryEmbedding(dim=64, max_seq_len=8192)\ncos, sin = rope(seq_len=2048, device='cuda')\n\n# In attention layer\nq_rotated, k_rotated = apply_rotary_pos_emb(query, key, cos, sin)\n```\n\n### ALiBi (Attention with Linear Biases)\n\n```python\ndef get_alibi_slopes(num_heads):\n    \"\"\"Get ALiBi slope values for each attention head.\"\"\"\n    def get_slopes_power_of_2(n):\n        start = 2 ** (-(2 ** -(math.log2(n) - 3)))\n        ratio = start\n        return [start * (ratio ** i) for i in range(n)]\n\n    if math.log2(num_heads).is_integer():\n        return get_slopes_power_of_2(num_heads)\n    else:\n        # Closest power of 2\n        closest_power = 2 ** math.floor(math.log2(num_heads))\n        slopes = get_slopes_power_of_2(closest_power)\n        # Add extra slopes\n        extra = get_slopes_power_of_2(2 * closest_power)\n        slopes.extend(extra[0::2][:num_heads - closest_power])\n        return slopes\n\ndef create_alibi_bias(seq_len, num_heads):\n    \"\"\"Create ALiBi attention bias.\"\"\"\n    # Distance matrix\n    context_position = torch.arange(seq_len)\n    memory_position = torch.arange(seq_len)\n    relative_position = memory_position[None, :] - context_position[:, None]\n\n    # Get slopes\n    slopes = torch.tensor(get_alibi_slopes(num_heads))\n\n    # Apply slopes to distances\n    alibi = slopes[:, None, None] * relative_position[None, :, :]\n    return alibi  # (num_heads, seq_len, seq_len)\n\n# Usage in attention\nnum_heads = 8\nseq_len = 2048\nalibi_bias = create_alibi_bias(seq_len, num_heads).to('cuda')\n\n# Add bias to attention scores\n# attn_scores shape: (batch, num_heads, seq_len, seq_len)\nattn_scores = attn_scores + alibi_bias\nattn_weights = torch.softmax(attn_scores, dim=-1)\n```\n\n### Position Interpolation for LLaMA\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\n# Original context: 2048 tokens\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Extend to 32k with position interpolation\n# Modify RoPE base frequency\nmodel.config.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 16.0  # 2048 * 16 = 32768\n}\n\n# Or use dynamic scaling\nmodel.config.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 16.0\n}\n\n# Fine-tune with long documents (minimal steps needed)\n# Position interpolation works out-of-the-box after this config change\n```\n\n## Core Concepts\n\n### 1. RoPE (Rotary Position Embeddings)\n\n**How it works:**\n- Encodes absolute position via rotation matrix\n- Provides relative position dependency in attention\n- Enables length extrapolation\n\n**Mathematical formulation:**\n```\nq_m = (W_q * x_m) * e^(imθ)\nk_n = (W_k * x_n) * e^(inθ)\n\nwhere θ_j = base^(-2j\u002Fd) for j ∈ [0, d\u002F2)\n```\n\n**Advantages:**\n- Decaying inter-token dependency with distance\n- Compatible with linear attention\n- Better extrapolation than absolute position encodings\n\n### 2. YaRN (Yet another RoPE extensioN)\n\n**Key innovation:**\n- NTK-aware interpolation (Neural Tangent Kernel)\n- Attention temperature scaling\n- Efficient context extension (10× less tokens vs baselines)\n\n**Parameters:**\n```python\n# YaRN configuration\nyarn_config = {\n    \"scale\": 16,                    # Extension factor\n    \"original_max_position\": 2048,  # Base context\n    \"extrapolation_factor\": 1.0,    # NTK parameter\n    \"attn_factor\": 1.0,             # Attention scaling\n    \"beta_fast\": 32,                # High-frequency scale\n    \"beta_slow\": 1,                 # Low-frequency scale\n}\n```\n\n**Performance:**\n- Extends LLaMA to 128k tokens\n- 2.5× less training steps than baselines\n- State-of-the-art context window extension\n\n### 3. ALiBi (Attention with Linear Biases)\n\n**Core idea:**\n- No positional embeddings added to tokens\n- Apply distance penalty directly to attention scores\n- Bias proportional to key-query distance\n\n**Formula:**\n```\nattention_bias[i, j] = -m * |i - j|\n\nwhere m = slope for each attention head\n```\n\n**Advantages:**\n- 11% faster training vs sinusoidal embeddings\n- 11% less memory usage\n- Strong length extrapolation (train 1k, test 2k+)\n- Inductive bias towards recency\n\n### 4. Position Interpolation\n\n**Technique:**\n- Linearly down-scale position indices\n- Interpolate within trained range (vs extrapolate beyond)\n- Minimal fine-tuning required\n\n**Formula:**\n```\n# Original: position indices [0, 1, 2, ..., L]\n# Extended: position indices [0, 0.5, 1.0, ..., L\u002F2]\n# (for 2× extension)\n\nscaled_position[i] = i \u002F extension_factor\n```\n\n**Results:**\n- LLaMA 7B-65B extended to 32k tokens\n- 1000 fine-tuning steps sufficient\n- 600× better stability than extrapolation\n\n## Method Comparison\n\n| Method | Max Context | Training Needed | Memory | Extrapolation | Best For |\n|--------|-------------|-----------------|--------|---------------|----------|\n| **RoPE** | 8k-32k | Full pre-training | Moderate | Good | New models |\n| **YaRN** | 32k-128k | Minimal (10× efficient) | Moderate | Excellent | Extending existing models |\n| **ALiBi** | Unlimited | Full pre-training | Low (-11%) | Excellent | Training from scratch |\n| **Position Interpolation** | 32k+ | Minimal (1k steps) | Moderate | Poor (by design) | Quick extension |\n\n## Implementation Patterns\n\n### HuggingFace Transformers Integration\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoConfig\n\n# RoPE with YaRN scaling\nconfig = AutoConfig.from_pretrained(\"mistralai\u002FMistral-7B-v0.1\")\nconfig.rope_scaling = {\n    \"type\": \"yarn\",\n    \"factor\": 8.0,\n    \"original_max_position_embeddings\": 8192,\n    \"attention_factor\": 1.0\n}\n\nmodel = AutoModelForCausalLM.from_config(config)\n\n# Position interpolation (simpler)\nconfig.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 4.0\n}\n\n# Dynamic scaling (adjusts based on input length)\nconfig.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 8.0\n}\n```\n\n### Custom RoPE Implementation\n\n```python\nclass LongContextAttention(nn.Module):\n    \"\"\"Multi-head attention with RoPE.\"\"\"\n\n    def __init__(self, hidden_size, num_heads, max_seq_len=32768):\n        super().__init__()\n        self.num_heads = num_heads\n        self.head_dim = hidden_size \u002F\u002F num_heads\n\n        # Q, K, V projections\n        self.q_proj = nn.Linear(hidden_size, hidden_size)\n        self.k_proj = nn.Linear(hidden_size, hidden_size)\n        self.v_proj = nn.Linear(hidden_size, hidden_size)\n        self.o_proj = nn.Linear(hidden_size, hidden_size)\n\n        # RoPE\n        self.rotary_emb = RotaryEmbedding(\n            dim=self.head_dim,\n            max_seq_len=max_seq_len\n        )\n\n    def forward(self, hidden_states):\n        batch_size, seq_len, _ = hidden_states.shape\n\n        # Project to Q, K, V\n        q = self.q_proj(hidden_states)\n        k = self.k_proj(hidden_states)\n        v = self.v_proj(hidden_states)\n\n        # Reshape for multi-head\n        q = q.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        k = k.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        v = v.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n\n        # Apply RoPE\n        cos, sin = self.rotary_emb(seq_len, device=hidden_states.device)\n        q, k = apply_rotary_pos_emb(q, k, cos, sin)\n\n        # Standard attention\n        attn_output = F.scaled_dot_product_attention(q, k, v)\n\n        # Reshape and project\n        attn_output = attn_output.transpose(1, 2).contiguous()\n        attn_output = attn_output.view(batch_size, seq_len, -1)\n        output = self.o_proj(attn_output)\n\n        return output\n```\n\n## Fine-tuning for Long Context\n\n### Minimal Fine-tuning (Position Interpolation)\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\n# Extend model config\nmodel.config.max_position_embeddings = 32768\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n\n# Training args (minimal steps needed)\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fllama-32k\",\n    num_train_epochs=1,\n    max_steps=1000,           # Only 1000 steps!\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n    learning_rate=2e-5,\n    warmup_steps=100,\n    logging_steps=10,\n    save_steps=500,\n)\n\n# Train on long documents\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=long_document_dataset,  # 32k token sequences\n)\n\ntrainer.train()\n```\n\n### YaRN Fine-tuning\n\n```bash\n# Clone YaRN implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\ncd yarn\n\n# Fine-tune LLaMA with YaRN\npython scripts\u002Ftrain.py \\\n    --model meta-llama\u002FLlama-2-7b-hf \\\n    --scale 16 \\\n    --rope_theta 10000 \\\n    --max_length 32768 \\\n    --batch_size 1 \\\n    --gradient_accumulation 16 \\\n    --steps 400 \\\n    --learning_rate 2e-5\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# For NEW models (training from scratch)\nuse_method = \"ALiBi\"  # Best extrapolation, lowest memory\n\n# For EXTENDING existing RoPE models\nuse_method = \"YaRN\"  # Most efficient extension (10× less data)\n\n# For QUICK extension with minimal compute\nuse_method = \"Position Interpolation\"  # 1000 steps\n\n# For MODERATE extension with good efficiency\nuse_method = \"Linear RoPE Scaling\"  # Built-in, simple\n```\n\n### 2. Scaling Factor Selection\n\n```python\n# Conservative (safer, better quality)\nscaling_factor = 2.0  # 8k → 16k\n\n# Moderate (good balance)\nscaling_factor = 4.0  # 8k → 32k\n\n# Aggressive (requires more fine-tuning)\nscaling_factor = 8.0  # 8k → 64k\nscaling_factor = 16.0  # 8k → 128k\n\n# Rule: Larger factors need more fine-tuning steps\nsteps_needed = 100 * scaling_factor  # Rough estimate\n```\n\n### 3. Fine-tuning Data\n\n```python\n# ✅ Good: Long documents matching target length\ntrain_data = [\n    {\"text\": long_doc_32k_tokens},  # Full 32k\n    {\"text\": long_doc_24k_tokens},  # Varied lengths\n    {\"text\": long_doc_16k_tokens},\n]\n\n# ❌ Bad: Short documents (won't learn long context)\ntrain_data = [\n    {\"text\": short_doc_2k_tokens},\n]\n\n# Use datasets like:\n# - PG-19 (books, long texts)\n# - arXiv papers\n# - Long-form conversations\n# - GitHub repositories (concatenated files)\n```\n\n### 4. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Applying position interpolation without fine-tuning\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n# Model will perform poorly without fine-tuning!\n\n# ✅ Good: Fine-tune after scaling\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\nfine_tune(model, long_documents, steps=1000)\n\n# ❌ Bad: Too aggressive scaling without data\nscale_to_1M_tokens()  # Won't work without massive fine-tuning\n\n# ✅ Good: Incremental scaling\n# 8k → 16k → 32k → 64k (fine-tune at each step)\n```\n\n## Production Deployment\n\n### Inference with Long Context\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load long-context model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"togethercomputer\u002FLLaMA-2-7B-32K\",  # 32k context\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"togethercomputer\u002FLLaMA-2-7B-32K\")\n\n# Process long document\nlong_text = \"...\" * 30000  # 30k tokens\ninputs = tokenizer(long_text, return_tensors=\"pt\", truncation=False).to('cuda')\n\n# Generate\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=512,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Memory Optimization\n\n```python\n# Use gradient checkpointing for fine-tuning\nmodel.gradient_checkpointing_enable()\n\n# Use Flash Attention 2\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    attn_implementation=\"flash_attention_2\",  # 2-3× faster\n    torch_dtype=torch.float16\n)\n\n# Use paged attention (vLLM)\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"togethercomputer\u002FLLaMA-2-7B-32K\",\n    max_model_len=32768,  # 32k context\n    gpu_memory_utilization=0.9\n)\n```\n\n## Resources\n\n- **RoPE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.09864 (RoFormer)\n- **YaRN Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.00071\n- **ALiBi Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2108.12409 (Train Short, Test Long)\n- **Position Interpolation**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.15595\n- **HuggingFace RoPE Utils**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fmain\u002Fsrc\u002Ftransformers\u002Fmodeling_rope_utils.py\n- **YaRN Implementation**: https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\n- **Together AI Blog**: https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fllama-2-7b-32k\n\n## See Also\n\n- `references\u002Frope.md` - Detailed RoPE implementation and theory\n- `references\u002Fextension_methods.md` - YaRN, ALiBi, Position Interpolation comparisons\n- `references\u002Ffine_tuning.md` - Complete fine-tuning guide for context extension\n\n\n",{"name":5,"tags":44,"author":54,"license":15,"version":55,"description":7,"dependencies":56},[25,45,24,46,47,48,49,50,51,52,53],"Long Context","YaRN","ALiBi","Position Interpolation","Extended Context","Rotary Embeddings","Attention Bias","Context Extension","Positional Encoding","Orchestra Research","1.0.0",[57,58,59],"transformers","torch","flash-attn","main","7925716f065257435fdb12e97f7e52842147d5ad",".codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-long-context","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude","https:\u002F\u002Fclaude.com\u002Fclaude-code","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho","Software · Applied AI · Quantitative Modeling · Building with Claude.","2026-06-10 12:24:46",{"source":8,"skillName":5,"commands":69,"prompt":79,"zipAvailable":80},[70,73,76],{"tool":71,"command":72},"npx","npx skills add henriquescastilho\u002Fmy-claude --skill long-context",{"tool":74,"command":75},"bunx","bunx skills add henriquescastilho\u002Fmy-claude --skill long-context",{"tool":77,"command":78},"pnpm","pnpm dlx skills add henriquescastilho\u002Fmy-claude --skill long-context","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-long-context\u002FSKILL.md",true,[82,88,92],{"path":83,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":86,"sourceHtmlUrl":87},"references\u002Fextension_methods.md","REFERENCE","参考文档",14537,"https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-long-context\u002Freferences\u002Fextension_methods.md",{"path":89,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":90,"sourceHtmlUrl":91},"references\u002Ffine_tuning.md",16431,"https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-long-context\u002Freferences\u002Ffine_tuning.md",{"path":93,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":94,"sourceHtmlUrl":95},"references\u002Frope.md",12121,"https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-long-context\u002Freferences\u002Frope.md",[97,138,147,156,178,205],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":119,"publishedAt":39},"01M6F0B6XPQ65AX2X0DWK8KT3E","markitdown","Convert files and office documents to Markdown. Supports PDF, DOCX, PPTX, XLSX, images (with OCR), audio (with transcription), HTML, CSV, JSON, XML, ZIP, YouTube URLs, EPubs and more.","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-writer\u002Fskills\u002Fmarkitdown\u002FSKILL.md",[103,108,111,114],{"categoryNo":104,"categoryName":105,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000002","文档处理","SCENARIO","使用场景",{"categoryNo":109,"categoryName":110,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000003","演示文稿",{"categoryNo":112,"categoryName":113,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000004","电子表格",{"categoryNo":115,"categoryName":116,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000034","行政与办公人员","PROFESSION","适用职业",[99,120,121,122,123,37,124,125,126,127,128,129,130,131,132,133,134,135,136,137],"docx","PDF","pptx","xlsx","csv","ocr","Audio","Bash","markdown","convert","read","write","edit","html","images","files","youtube","with",{"skillNo":139,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":140,"categories":141,"keywords":146,"publishedAt":39},"01M6F06SBHR9F0SDWE282Y8075","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-writer\u002Fscientific_writer\u002F.claude\u002Fskills\u002Fmarkitdown\u002FSKILL.md",[142,143,144,145],{"categoryNo":104,"categoryName":105,"taxonomyType":106,"taxonomyTypeDesc":107},{"categoryNo":109,"categoryName":110,"taxonomyType":106,"taxonomyTypeDesc":107},{"categoryNo":112,"categoryName":113,"taxonomyType":106,"taxonomyTypeDesc":107},{"categoryNo":115,"categoryName":116,"taxonomyType":117,"taxonomyTypeDesc":118},[99,120,121,122,123,37,124,125,126,127,128,129,130,131,132,133,134,135,136,137],{"skillNo":148,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":155,"publishedAt":39},"01M6F02D4R14EJAPBWVDF9J9VQ","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-writer\u002F.claude\u002Fskills\u002Fmarkitdown\u002FSKILL.md",[151,152,153,154],{"categoryNo":104,"categoryName":105,"taxonomyType":106,"taxonomyTypeDesc":107},{"categoryNo":109,"categoryName":110,"taxonomyType":106,"taxonomyTypeDesc":107},{"categoryNo":112,"categoryName":113,"taxonomyType":106,"taxonomyTypeDesc":107},{"categoryNo":115,"categoryName":116,"taxonomyType":117,"taxonomyTypeDesc":118},[99,120,121,122,123,37,124,125,126,127,128,129,130,131,132,133,134,135,136,137],{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":160,"categories":161,"keywords":162,"publishedAt":39},"01M6F000KS6YB4MKS774G4R8ND","zarr-python","Chunked N-D arrays for cloud storage. Compressed arrays, parallel I\u002FO, S3\u002FGCS integration, NumPy\u002FDask\u002FXarray compatible, for large-scale scientific computing pipelines.","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-skills\u002Fscientific-skills\u002Fzarr-python\u002FSKILL.md",[],[163,164,165,166,167,168,169,170,171,172,173,174,175,176,177],"python","zarr python","storage","integration","s3","gcs","cloud","dask","parallel","pipelines","arrays","numpy","scientific computing","scale","scientific",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":182,"categories":183,"keywords":187,"publishedAt":39},"01M6EZZJWFXM1GR78KTW6J0DN8","what-if-oracle","Run structured What-If scenario analysis with multi-branch possibility exploration. Use this skill when the user asks speculative questions like \"what if...\", \"what would happen if...\", \"what are the possibilities\", \"explore scenarios\", \"scenario analysis\", \"possibility space\", \"what could go wrong\", \"best case \u002F worst case\", \"risk analysis\", \"contingency planning\", \"strategic options\", or any question about uncertain futures. Also trigger when the user faces a fork-in-the-road decision, wants to stress-test an idea, or needs to think through consequences before committing.","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-skills\u002Fscientific-skills\u002Fwhat-if-oracle\u002FSKILL.md",[184],{"categoryNo":185,"categoryName":186,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000023","Skill与智能体开发",[188,189,190,191,192,193,194,195,196,197,198,130,131,199,200,201,202,22,203,204],"oracle","what if oracle","go","run","ux","futures","analysis","explore","decision","branch","question","idea","exploration","options","stress-test","scenario","stress",{"skillNo":206,"name":207,"nameZh":6,"description":208,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":209,"categories":210,"keywords":221,"publishedAt":39},"01M6EZZCXAEB49RXGQCJX5E8GB","venue-templates","Access comprehensive LaTeX templates, formatting requirements, and submission guidelines for major scientific publication venues (Nature, Science, PLOS, IEEE, ACM), academic conferences (NeurIPS, ICML, CVPR, CHI), research posters, and grant proposals (NSF, NIH, DOE, DARPA). This skill should be used when preparing manuscripts for journal submission, conference papers, research posters, or grant proposals and need venue-specific formatting requirements and templates.","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-skills\u002Fscientific-skills\u002Fvenue-templates\u002FSKILL.md",[211,214,217,218],{"categoryNo":212,"categoryName":213,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":215,"categoryName":216,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":185,"categoryName":186,"taxonomyType":106,"taxonomyTypeDesc":107},{"categoryNo":219,"categoryName":220,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000036","内容创作者",[222,223,224,225,226,227,228,229,127,230,130,131,132,231,232,233,234,235,177,236],"templates","venue templates","research","guidelines","NeurIPS","ICML","LaTeX","science","access","proposals","Academic","requirements","formatting","journal","chi"]