[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M74YYCM0Q7SBPQ8G3PT0AVVC":3,"skill-same:01M74YYCM0Q7SBPQ8G3PT0AVVC":72},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":30,"commitSha":31,"skillDirectory":32,"forksCount":33,"repositoryHtmlUrl":34,"repositoryHomepageUrl":35,"ownerHtmlUrl":36,"ownerBlogUrl":37,"ownerDescription":38,"updatedAt":39,"install":40,"resources":53},"01M74YYCM0Q7SBPQ8G3PT0AVVC","long-context",null,"Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.","KalarisLabs\u002Fresearch-agent-skills","KalarisLabs","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F300878568?v=4","01M74Y7MT23GXXC56VM6TVSXZY","Kalarislabs","01M74Y7MTQM9REXZX4SNXWX95C",132,"MIT","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Flong-context\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: long-context\ndescription: Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: multimodal-and-emerging\n  maintainer: Kalaris Labs\n  tags: Emerging Techniques, Long Context, RoPE, YaRN, ALiBi, Position Interpolation, Extended Context, Rotary Embeddings, Attention Bias, Context Extension, Positional Encoding\n  dependencies: transformers, torch, flash-attn\n---\n\n# Long Context: Extending Transformer Context Windows\n\n## When to Use This Skill\n\nUse Long Context techniques when you need to:\n- **Process long documents** (32k, 64k, 128k+ tokens) with transformer models\n- **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.)\n- **Implement efficient positional encodings** (RoPE, ALiBi)\n- **Train models** with length extrapolation capabilities\n- **Deploy models** that handle variable-length inputs efficiently\n- **Fine-tune** existing models for longer contexts with minimal compute\n\n**Key Techniques**: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation\n\n**Papers**: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595)\n\n## Installation\n\n```bash\n# HuggingFace Transformers (includes RoPE, YaRN support)\npip install transformers torch\n\n# For custom implementations\npip install einops  # Tensor operations\npip install rotary-embedding-torch  # Standalone RoPE\n\n# Optional: FlashAttention for efficiency\npip install flash-attn --no-build-isolation\n```\n\n## Quick Start\n\n### RoPE (Rotary Position Embeddings)\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass RotaryEmbedding(nn.Module):\n    \"\"\"Rotary Position Embeddings (RoPE).\"\"\"\n\n    def __init__(self, dim, max_seq_len=8192, base=10000):\n        super().__init__()\n        # Compute inverse frequencies\n        inv_freq = 1.0 \u002F (base ** (torch.arange(0, dim, 2).float() \u002F dim))\n        self.register_buffer(\"inv_freq\", inv_freq)\n        self.max_seq_len = max_seq_len\n\n    def forward(self, seq_len, device):\n        # Position indices\n        t = torch.arange(seq_len, device=device).type_as(self.inv_freq)\n\n        # Compute frequencies\n        freqs = torch.outer(t, self.inv_freq)  # (seq_len, dim\u002F2)\n\n        # Compute sin and cos\n        emb = torch.cat((freqs, freqs), dim=-1)  # (seq_len, dim)\n        return emb.cos(), emb.sin()\n\ndef rotate_half(x):\n    \"\"\"Rotate half the hidden dimensions.\"\"\"\n    x1, x2 = x.chunk(2, dim=-1)\n    return torch.cat((-x2, x1), dim=-1)\n\ndef apply_rotary_pos_emb(q, k, cos, sin):\n    \"\"\"Apply rotary embeddings to queries and keys.\"\"\"\n    # q, k shape: (batch, heads, seq_len, dim)\n    q_embed = (q * cos) + (rotate_half(q) * sin)\n    k_embed = (k * cos) + (rotate_half(k) * sin)\n    return q_embed, k_embed\n\n# Usage\nrope = RotaryEmbedding(dim=64, max_seq_len=8192)\ncos, sin = rope(seq_len=2048, device='cuda')\n\n# In attention layer\nq_rotated, k_rotated = apply_rotary_pos_emb(query, key, cos, sin)\n```\n\n### ALiBi (Attention with Linear Biases)\n\n```python\ndef get_alibi_slopes(num_heads):\n    \"\"\"Get ALiBi slope values for each attention head.\"\"\"\n    def get_slopes_power_of_2(n):\n        start = 2 ** (-(2 ** -(math.log2(n) - 3)))\n        ratio = start\n        return [start * (ratio ** i) for i in range(n)]\n\n    if math.log2(num_heads).is_integer():\n        return get_slopes_power_of_2(num_heads)\n    else:\n        # Closest power of 2\n        closest_power = 2 ** math.floor(math.log2(num_heads))\n        slopes = get_slopes_power_of_2(closest_power)\n        # Add extra slopes\n        extra = get_slopes_power_of_2(2 * closest_power)\n        slopes.extend(extra[0::2][:num_heads - closest_power])\n        return slopes\n\ndef create_alibi_bias(seq_len, num_heads):\n    \"\"\"Create ALiBi attention bias.\"\"\"\n    # Distance matrix\n    context_position = torch.arange(seq_len)\n    memory_position = torch.arange(seq_len)\n    relative_position = memory_position[None, :] - context_position[:, None]\n\n    # Get slopes\n    slopes = torch.tensor(get_alibi_slopes(num_heads))\n\n    # Apply slopes to distances\n    alibi = slopes[:, None, None] * relative_position[None, :, :]\n    return alibi  # (num_heads, seq_len, seq_len)\n\n# Usage in attention\nnum_heads = 8\nseq_len = 2048\nalibi_bias = create_alibi_bias(seq_len, num_heads).to('cuda')\n\n# Add bias to attention scores\n# attn_scores shape: (batch, num_heads, seq_len, seq_len)\nattn_scores = attn_scores + alibi_bias\nattn_weights = torch.softmax(attn_scores, dim=-1)\n```\n\n### Position Interpolation for LLaMA\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\n# Original context: 2048 tokens\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Extend to 32k with position interpolation\n# Modify RoPE base frequency\nmodel.config.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 16.0  # 2048 * 16 = 32768\n}\n\n# Or use dynamic scaling\nmodel.config.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 16.0\n}\n\n# Fine-tune with long documents (minimal steps needed)\n# Position interpolation works out-of-the-box after this config change\n```\n\n## Core Concepts\n\n### 1. RoPE (Rotary Position Embeddings)\n\n**How it works:**\n- Encodes absolute position via rotation matrix\n- Provides relative position dependency in attention\n- Enables length extrapolation\n\n**Mathematical formulation:**\n```\nq_m = (W_q * x_m) * e^(imθ)\nk_n = (W_k * x_n) * e^(inθ)\n\nwhere θ_j = base^(-2j\u002Fd) for j ∈ [0, d\u002F2)\n```\n\n**Advantages:**\n- Decaying inter-token dependency with distance\n- Compatible with linear attention\n- Better extrapolation than absolute position encodings\n\n### 2. YaRN (Yet another RoPE extensioN)\n\n**Key innovation:**\n- NTK-aware interpolation (Neural Tangent Kernel)\n- Attention temperature scaling\n- Efficient context extension (10× less tokens vs baselines)\n\n**Parameters:**\n```python\n# YaRN configuration\nyarn_config = {\n    \"scale\": 16,                    # Extension factor\n    \"original_max_position\": 2048,  # Base context\n    \"extrapolation_factor\": 1.0,    # NTK parameter\n    \"attn_factor\": 1.0,             # Attention scaling\n    \"beta_fast\": 32,                # High-frequency scale\n    \"beta_slow\": 1,                 # Low-frequency scale\n}\n```\n\n**Performance:**\n- Extends LLaMA to 128k tokens\n- 2.5× less training steps than baselines\n- State-of-the-art context window extension\n\n### 3. ALiBi (Attention with Linear Biases)\n\n**Core idea:**\n- No positional embeddings added to tokens\n- Apply distance penalty directly to attention scores\n- Bias proportional to key-query distance\n\n**Formula:**\n```\nattention_bias[i, j] = -m * |i - j|\n\nwhere m = slope for each attention head\n```\n\n**Advantages:**\n- 11% faster training vs sinusoidal embeddings\n- 11% less memory usage\n- Strong length extrapolation (train 1k, test 2k+)\n- Inductive bias towards recency\n\n### 4. Position Interpolation\n\n**Technique:**\n- Linearly down-scale position indices\n- Interpolate within trained range (vs extrapolate beyond)\n- Minimal fine-tuning required\n\n**Formula:**\n```\n# Original: position indices [0, 1, 2, ..., L]\n# Extended: position indices [0, 0.5, 1.0, ..., L\u002F2]\n# (for 2× extension)\n\nscaled_position[i] = i \u002F extension_factor\n```\n\n**Results:**\n- LLaMA 7B-65B extended to 32k tokens\n- 1000 fine-tuning steps sufficient\n- 600× better stability than extrapolation\n\n## Method Comparison\n\n| Method | Max Context | Training Needed | Memory | Extrapolation | Best For |\n|--------|-------------|-----------------|--------|---------------|----------|\n| **RoPE** | 8k-32k | Full pre-training | Moderate | Good | New models |\n| **YaRN** | 32k-128k | Minimal (10× efficient) | Moderate | Excellent | Extending existing models |\n| **ALiBi** | Unlimited | Full pre-training | Low (-11%) | Excellent | Training from scratch |\n| **Position Interpolation** | 32k+ | Minimal (1k steps) | Moderate | Poor (by design) | Quick extension |\n\n## Implementation Patterns\n\nDetails, code examples and parameter tables: [references\u002Fimplementation-patterns.md](references\u002Fimplementation-patterns.md). Read it when this step applies.\n\n## Fine-tuning for Long Context\n\n### Minimal Fine-tuning (Position Interpolation)\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\n# Extend model config\nmodel.config.max_position_embeddings = 32768\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n\n# Training args (minimal steps needed)\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fllama-32k\",\n    num_train_epochs=1,\n    max_steps=1000,           # Only 1000 steps!\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n    learning_rate=2e-5,\n    warmup_steps=100,\n    logging_steps=10,\n    save_steps=500,\n)\n\n# Train on long documents\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=long_document_dataset,  # 32k token sequences\n)\n\ntrainer.train()\n```\n\n### YaRN Fine-tuning\n\n```bash\n# Clone YaRN implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\ncd yarn\n\n# Fine-tune LLaMA with YaRN\npython scripts\u002Ftrain.py \\\n    --model meta-llama\u002FLlama-2-7b-hf \\\n    --scale 16 \\\n    --rope_theta 10000 \\\n    --max_length 32768 \\\n    --batch_size 1 \\\n    --gradient_accumulation 16 \\\n    --steps 400 \\\n    --learning_rate 2e-5\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# For NEW models (training from scratch)\nuse_method = \"ALiBi\"  # Best extrapolation, lowest memory\n\n# For EXTENDING existing RoPE models\nuse_method = \"YaRN\"  # Most efficient extension (10× less data)\n\n# For QUICK extension with minimal compute\nuse_method = \"Position Interpolation\"  # 1000 steps\n\n# For MODERATE extension with good efficiency\nuse_method = \"Linear RoPE Scaling\"  # Built-in, simple\n```\n\n### 2. Scaling Factor Selection\n\n```python\n# Conservative (safer, better quality)\nscaling_factor = 2.0  # 8k → 16k\n\n# Moderate (good balance)\nscaling_factor = 4.0  # 8k → 32k\n\n# Aggressive (requires more fine-tuning)\nscaling_factor = 8.0  # 8k → 64k\nscaling_factor = 16.0  # 8k → 128k\n\n# Rule: Larger factors need more fine-tuning steps\nsteps_needed = 100 * scaling_factor  # Rough estimate\n```\n\n### 3. Fine-tuning Data\n\n```python\n# ✅ Good: Long documents matching target length\ntrain_data = [\n    {\"text\": long_doc_32k_tokens},  # Full 32k\n    {\"text\": long_doc_24k_tokens},  # Varied lengths\n    {\"text\": long_doc_16k_tokens},\n]\n\n# ❌ Bad: Short documents (won't learn long context)\ntrain_data = [\n    {\"text\": short_doc_2k_tokens},\n]\n\n# Use datasets like:\n# - PG-19 (books, long texts)\n# - arXiv papers\n# - Long-form conversations\n# - GitHub repositories (concatenated files)\n```\n\n### 4. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Applying position interpolation without fine-tuning\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n# Model will perform poorly without fine-tuning!\n\n# ✅ Good: Fine-tune after scaling\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\nfine_tune(model, long_documents, steps=1000)\n\n# ❌ Bad: Too aggressive scaling without data\nscale_to_1M_tokens()  # Won't work without massive fine-tuning\n\n# ✅ Good: Incremental scaling\n# 8k → 16k → 32k → 64k (fine-tune at each step)\n```\n\n## Production Deployment\n\n### Inference with Long Context\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load long-context model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"togethercomputer\u002FLLaMA-2-7B-32K\",  # 32k context\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"togethercomputer\u002FLLaMA-2-7B-32K\")\n\n# Process long document\nlong_text = \"...\" * 30000  # 30k tokens\ninputs = tokenizer(long_text, return_tensors=\"pt\", truncation=False).to('cuda')\n\n# Generate\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=512,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Memory Optimization\n\n```python\n# Use gradient checkpointing for fine-tuning\nmodel.gradient_checkpointing_enable()\n\n# Use Flash Attention 2\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    attn_implementation=\"flash_attention_2\",  # 2-3× faster\n    torch_dtype=torch.float16\n)\n\n# Use paged attention (vLLM)\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"togethercomputer\u002FLLaMA-2-7B-32K\",\n    max_model_len=32768,  # 32k context\n    gpu_memory_utilization=0.9\n)\n```\n\n## Resources\n\n- **RoPE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.09864 (RoFormer)\n- **YaRN Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.00071\n- **ALiBi Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2108.12409 (Train Short, Test Long)\n- **Position Interpolation**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.15595\n- **HuggingFace RoPE Utils**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fmain\u002Fsrc\u002Ftransformers\u002Fmodeling_rope_utils.py\n- **YaRN Implementation**: https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\n- **Together AI Blog**: https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fllama-2-7b-32k\n\n## See Also\n\n- `references\u002Frope.md` - Detailed RoPE implementation and theory\n- `references\u002Fextension_methods.md` - YaRN, ALiBi, Position Interpolation comparisons\n- `references\u002Ffine_tuning.md` - Complete fine-tuning guide for context extension\n\n## Agent operating procedure\n\n1. **Check the environment.** Confirm GPU memory, model checkpoint and licence, framework versions and input formats.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Run inference on a single sample at low resolution or short length first.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Inspect outputs qualitatively and with task metrics; check licence restrictions for generated content.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| Out-of-memory on large inputs | Reduce resolution or sequence length, use half precision, or tile the input. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Check model and dataset licences before commercial or published use.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n","# Long Context: Extending Transformer Context Windows\n\n## When to Use This Skill\n\nUse Long Context techniques when you need to:\n- **Process long documents** (32k, 64k, 128k+ tokens) with transformer models\n- **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.)\n- **Implement efficient positional encodings** (RoPE, ALiBi)\n- **Train models** with length extrapolation capabilities\n- **Deploy models** that handle variable-length inputs efficiently\n- **Fine-tune** existing models for longer contexts with minimal compute\n\n**Key Techniques**: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation\n\n**Papers**: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595)\n\n## Installation\n\n```bash\n# HuggingFace Transformers (includes RoPE, YaRN support)\npip install transformers torch\n\n# For custom implementations\npip install einops  # Tensor operations\npip install rotary-embedding-torch  # Standalone RoPE\n\n# Optional: FlashAttention for efficiency\npip install flash-attn --no-build-isolation\n```\n\n## Quick Start\n\n### RoPE (Rotary Position Embeddings)\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass RotaryEmbedding(nn.Module):\n    \"\"\"Rotary Position Embeddings (RoPE).\"\"\"\n\n    def __init__(self, dim, max_seq_len=8192, base=10000):\n        super().__init__()\n        # Compute inverse frequencies\n        inv_freq = 1.0 \u002F (base ** (torch.arange(0, dim, 2).float() \u002F dim))\n        self.register_buffer(\"inv_freq\", inv_freq)\n        self.max_seq_len = max_seq_len\n\n    def forward(self, seq_len, device):\n        # Position indices\n        t = torch.arange(seq_len, device=device).type_as(self.inv_freq)\n\n        # Compute frequencies\n        freqs = torch.outer(t, self.inv_freq)  # (seq_len, dim\u002F2)\n\n        # Compute sin and cos\n        emb = torch.cat((freqs, freqs), dim=-1)  # (seq_len, dim)\n        return emb.cos(), emb.sin()\n\ndef rotate_half(x):\n    \"\"\"Rotate half the hidden dimensions.\"\"\"\n    x1, x2 = x.chunk(2, dim=-1)\n    return torch.cat((-x2, x1), dim=-1)\n\ndef apply_rotary_pos_emb(q, k, cos, sin):\n    \"\"\"Apply rotary embeddings to queries and keys.\"\"\"\n    # q, k shape: (batch, heads, seq_len, dim)\n    q_embed = (q * cos) + (rotate_half(q) * sin)\n    k_embed = (k * cos) + (rotate_half(k) * sin)\n    return q_embed, k_embed\n\n# Usage\nrope = RotaryEmbedding(dim=64, max_seq_len=8192)\ncos, sin = rope(seq_len=2048, device='cuda')\n\n# In attention layer\nq_rotated, k_rotated = apply_rotary_pos_emb(query, key, cos, sin)\n```\n\n### ALiBi (Attention with Linear Biases)\n\n```python\ndef get_alibi_slopes(num_heads):\n    \"\"\"Get ALiBi slope values for each attention head.\"\"\"\n    def get_slopes_power_of_2(n):\n        start = 2 ** (-(2 ** -(math.log2(n) - 3)))\n        ratio = start\n        return [start * (ratio ** i) for i in range(n)]\n\n    if math.log2(num_heads).is_integer():\n        return get_slopes_power_of_2(num_heads)\n    else:\n        # Closest power of 2\n        closest_power = 2 ** math.floor(math.log2(num_heads))\n        slopes = get_slopes_power_of_2(closest_power)\n        # Add extra slopes\n        extra = get_slopes_power_of_2(2 * closest_power)\n        slopes.extend(extra[0::2][:num_heads - closest_power])\n        return slopes\n\ndef create_alibi_bias(seq_len, num_heads):\n    \"\"\"Create ALiBi attention bias.\"\"\"\n    # Distance matrix\n    context_position = torch.arange(seq_len)\n    memory_position = torch.arange(seq_len)\n    relative_position = memory_position[None, :] - context_position[:, None]\n\n    # Get slopes\n    slopes = torch.tensor(get_alibi_slopes(num_heads))\n\n    # Apply slopes to distances\n    alibi = slopes[:, None, None] * relative_position[None, :, :]\n    return alibi  # (num_heads, seq_len, seq_len)\n\n# Usage in attention\nnum_heads = 8\nseq_len = 2048\nalibi_bias = create_alibi_bias(seq_len, num_heads).to('cuda')\n\n# Add bias to attention scores\n# attn_scores shape: (batch, num_heads, seq_len, seq_len)\nattn_scores = attn_scores + alibi_bias\nattn_weights = torch.softmax(attn_scores, dim=-1)\n```\n\n### Position Interpolation for LLaMA\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\n# Original context: 2048 tokens\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Extend to 32k with position interpolation\n# Modify RoPE base frequency\nmodel.config.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 16.0  # 2048 * 16 = 32768\n}\n\n# Or use dynamic scaling\nmodel.config.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 16.0\n}\n\n# Fine-tune with long documents (minimal steps needed)\n# Position interpolation works out-of-the-box after this config change\n```\n\n## Core Concepts\n\n### 1. RoPE (Rotary Position Embeddings)\n\n**How it works:**\n- Encodes absolute position via rotation matrix\n- Provides relative position dependency in attention\n- Enables length extrapolation\n\n**Mathematical formulation:**\n```\nq_m = (W_q * x_m) * e^(imθ)\nk_n = (W_k * x_n) * e^(inθ)\n\nwhere θ_j = base^(-2j\u002Fd) for j ∈ [0, d\u002F2)\n```\n\n**Advantages:**\n- Decaying inter-token dependency with distance\n- Compatible with linear attention\n- Better extrapolation than absolute position encodings\n\n### 2. YaRN (Yet another RoPE extensioN)\n\n**Key innovation:**\n- NTK-aware interpolation (Neural Tangent Kernel)\n- Attention temperature scaling\n- Efficient context extension (10× less tokens vs baselines)\n\n**Parameters:**\n```python\n# YaRN configuration\nyarn_config = {\n    \"scale\": 16,                    # Extension factor\n    \"original_max_position\": 2048,  # Base context\n    \"extrapolation_factor\": 1.0,    # NTK parameter\n    \"attn_factor\": 1.0,             # Attention scaling\n    \"beta_fast\": 32,                # High-frequency scale\n    \"beta_slow\": 1,                 # Low-frequency scale\n}\n```\n\n**Performance:**\n- Extends LLaMA to 128k tokens\n- 2.5× less training steps than baselines\n- State-of-the-art context window extension\n\n### 3. ALiBi (Attention with Linear Biases)\n\n**Core idea:**\n- No positional embeddings added to tokens\n- Apply distance penalty directly to attention scores\n- Bias proportional to key-query distance\n\n**Formula:**\n```\nattention_bias[i, j] = -m * |i - j|\n\nwhere m = slope for each attention head\n```\n\n**Advantages:**\n- 11% faster training vs sinusoidal embeddings\n- 11% less memory usage\n- Strong length extrapolation (train 1k, test 2k+)\n- Inductive bias towards recency\n\n### 4. Position Interpolation\n\n**Technique:**\n- Linearly down-scale position indices\n- Interpolate within trained range (vs extrapolate beyond)\n- Minimal fine-tuning required\n\n**Formula:**\n```\n# Original: position indices [0, 1, 2, ..., L]\n# Extended: position indices [0, 0.5, 1.0, ..., L\u002F2]\n# (for 2× extension)\n\nscaled_position[i] = i \u002F extension_factor\n```\n\n**Results:**\n- LLaMA 7B-65B extended to 32k tokens\n- 1000 fine-tuning steps sufficient\n- 600× better stability than extrapolation\n\n## Method Comparison\n\n| Method | Max Context | Training Needed | Memory | Extrapolation | Best For |\n|--------|-------------|-----------------|--------|---------------|----------|\n| **RoPE** | 8k-32k | Full pre-training | Moderate | Good | New models |\n| **YaRN** | 32k-128k | Minimal (10× efficient) | Moderate | Excellent | Extending existing models |\n| **ALiBi** | Unlimited | Full pre-training | Low (-11%) | Excellent | Training from scratch |\n| **Position Interpolation** | 32k+ | Minimal (1k steps) | Moderate | Poor (by design) | Quick extension |\n\n## Implementation Patterns\n\nDetails, code examples and parameter tables: [references\u002Fimplementation-patterns.md](references\u002Fimplementation-patterns.md). Read it when this step applies.\n\n## Fine-tuning for Long Context\n\n### Minimal Fine-tuning (Position Interpolation)\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\n# Extend model config\nmodel.config.max_position_embeddings = 32768\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n\n# Training args (minimal steps needed)\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fllama-32k\",\n    num_train_epochs=1,\n    max_steps=1000,           # Only 1000 steps!\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n    learning_rate=2e-5,\n    warmup_steps=100,\n    logging_steps=10,\n    save_steps=500,\n)\n\n# Train on long documents\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=long_document_dataset,  # 32k token sequences\n)\n\ntrainer.train()\n```\n\n### YaRN Fine-tuning\n\n```bash\n# Clone YaRN implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\ncd yarn\n\n# Fine-tune LLaMA with YaRN\npython scripts\u002Ftrain.py \\\n    --model meta-llama\u002FLlama-2-7b-hf \\\n    --scale 16 \\\n    --rope_theta 10000 \\\n    --max_length 32768 \\\n    --batch_size 1 \\\n    --gradient_accumulation 16 \\\n    --steps 400 \\\n    --learning_rate 2e-5\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# For NEW models (training from scratch)\nuse_method = \"ALiBi\"  # Best extrapolation, lowest memory\n\n# For EXTENDING existing RoPE models\nuse_method = \"YaRN\"  # Most efficient extension (10× less data)\n\n# For QUICK extension with minimal compute\nuse_method = \"Position Interpolation\"  # 1000 steps\n\n# For MODERATE extension with good efficiency\nuse_method = \"Linear RoPE Scaling\"  # Built-in, simple\n```\n\n### 2. Scaling Factor Selection\n\n```python\n# Conservative (safer, better quality)\nscaling_factor = 2.0  # 8k → 16k\n\n# Moderate (good balance)\nscaling_factor = 4.0  # 8k → 32k\n\n# Aggressive (requires more fine-tuning)\nscaling_factor = 8.0  # 8k → 64k\nscaling_factor = 16.0  # 8k → 128k\n\n# Rule: Larger factors need more fine-tuning steps\nsteps_needed = 100 * scaling_factor  # Rough estimate\n```\n\n### 3. Fine-tuning Data\n\n```python\n# ✅ Good: Long documents matching target length\ntrain_data = [\n    {\"text\": long_doc_32k_tokens},  # Full 32k\n    {\"text\": long_doc_24k_tokens},  # Varied lengths\n    {\"text\": long_doc_16k_tokens},\n]\n\n# ❌ Bad: Short documents (won't learn long context)\ntrain_data = [\n    {\"text\": short_doc_2k_tokens},\n]\n\n# Use datasets like:\n# - PG-19 (books, long texts)\n# - arXiv papers\n# - Long-form conversations\n# - GitHub repositories (concatenated files)\n```\n\n### 4. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Applying position interpolation without fine-tuning\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n# Model will perform poorly without fine-tuning!\n\n# ✅ Good: Fine-tune after scaling\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\nfine_tune(model, long_documents, steps=1000)\n\n# ❌ Bad: Too aggressive scaling without data\nscale_to_1M_tokens()  # Won't work without massive fine-tuning\n\n# ✅ Good: Incremental scaling\n# 8k → 16k → 32k → 64k (fine-tune at each step)\n```\n\n## Production Deployment\n\n### Inference with Long Context\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load long-context model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"togethercomputer\u002FLLaMA-2-7B-32K\",  # 32k context\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"togethercomputer\u002FLLaMA-2-7B-32K\")\n\n# Process long document\nlong_text = \"...\" * 30000  # 30k tokens\ninputs = tokenizer(long_text, return_tensors=\"pt\", truncation=False).to('cuda')\n\n# Generate\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=512,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Memory Optimization\n\n```python\n# Use gradient checkpointing for fine-tuning\nmodel.gradient_checkpointing_enable()\n\n# Use Flash Attention 2\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    attn_implementation=\"flash_attention_2\",  # 2-3× faster\n    torch_dtype=torch.float16\n)\n\n# Use paged attention (vLLM)\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"togethercomputer\u002FLLaMA-2-7B-32K\",\n    max_model_len=32768,  # 32k context\n    gpu_memory_utilization=0.9\n)\n```\n\n## Resources\n\n- **RoPE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.09864 (RoFormer)\n- **YaRN Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.00071\n- **ALiBi Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2108.12409 (Train Short, Test Long)\n- **Position Interpolation**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.15595\n- **HuggingFace RoPE Utils**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fmain\u002Fsrc\u002Ftransformers\u002Fmodeling_rope_utils.py\n- **YaRN Implementation**: https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\n- **Together AI Blog**: https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fllama-2-7b-32k\n\n## See Also\n\n- `references\u002Frope.md` - Detailed RoPE implementation and theory\n- `references\u002Fextension_methods.md` - YaRN, ALiBi, Position Interpolation comparisons\n- `references\u002Ffine_tuning.md` - Complete fine-tuning guide for context extension\n\n## Agent operating procedure\n\n1. **Check the environment.** Confirm GPU memory, model checkpoint and licence, framework versions and input formats.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Run inference on a single sample at low resolution or short length first.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Inspect outputs qualitatively and with task metrics; check licence restrictions for generated content.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| Out-of-memory on large inputs | Reduce resolution or sequence length, use half precision, or tile the input. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Check model and dataset licences before commercial or published use.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n",{"name":5,"license":15,"metadata":24,"description":7},{"tags":25,"version":26,"category":27,"maintainer":28,"dependencies":29},"Emerging Techniques, Long Context, RoPE, YaRN, ALiBi, Position Interpolation, Extended Context, Rotary Embeddings, Attention Bias, Context Extension, Positional Encoding","1.0.0","multimodal-and-emerging","Kalaris Labs","transformers, torch, flash-attn","main","29fe36d26127205057d5e0ed8425f6b0e9a8c13a","skills\u002Flong-context",16,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills","https:\u002F\u002Fdocs.kalarislabs.com\u002Fresearch-agent-skills","https:\u002F\u002Fgithub.com\u002FKalarisLabs","kalarislabs.com","Applied AI lab building the infrastructure for the agentic era","2026-11-14 17:49:08",{"source":8,"skillName":5,"commands":41,"prompt":51,"zipAvailable":52},[42,45,48],{"tool":43,"command":44},"npx","npx skills add KalarisLabs\u002Fresearch-agent-skills --skill long-context",{"tool":46,"command":47},"bunx","bunx skills add KalarisLabs\u002Fresearch-agent-skills --skill long-context",{"tool":49,"command":50},"pnpm","pnpm dlx skills add KalarisLabs\u002Fresearch-agent-skills --skill long-context","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Flong-context\u002FSKILL.md",true,[54,60,64,68],{"path":55,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":58,"sourceHtmlUrl":59},"references\u002Fextension_methods.md","REFERENCE","参考文档",14537,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Flong-context\u002Freferences\u002Fextension_methods.md",{"path":61,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Ffine_tuning.md",16431,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Flong-context\u002Freferences\u002Ffine_tuning.md",{"path":65,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fimplementation-patterns.md",2343,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Flong-context\u002Freferences\u002Fimplementation-patterns.md",{"path":69,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Frope.md",12121,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Flong-context\u002Freferences\u002Frope.md",[73,80,87,94,101,108],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":19},"01M74Z8MGT305R11SNY5G28QH5","plos","Prepare manuscripts for PLOS journals (PLOS ONE, PLOS Biology, PLOS Computational Biology, PLOS Genetics, PLOS Medicine, PLOS Pathogens, PLOS Neglected Tropical Diseases, PLOS Global Public Health and others), covering structure, abstract and author summary, Vancouver-style references, the mandatory data availability policy, ethics and financial disclosure statements, figure requirements (PACE), the PLOS LaTeX template, reporting guidelines and PLOS ONE's publication criteria. Use when targeting any PLOS journal or checking compliance with PLOS policies.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fplos\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M74Z8CCN9APXYHNQ95V1EBGT","pkpd-modeling","Pharmacokinetic and pharmacodynamic modelling and simulation - non-compartmental analysis, compartmental and population PK, PK\u002FPD and exposure-response, TMDD, PBPK orientation, bioequivalence, allometric scaling and first-in-human dose, drug interaction prediction, and Bayesian therapeutic drug monitoring. Use when analysing concentration-time data, deriving exposure metrics, fitting PK or PD models, or evaluating dosing regimens. Triggers include \"pharmacokinetics\", \"pharmacodynamics\", \"PK\u002FPD\", \"NCA\", \"non-compartmental\", \"AUC\", \"Cmax\", \"lambda z\", \"half-life\", \"clearance\", \"volume of distribution\", \"compartmental model\", \"population PK\", \"popPK\", \"NONMEM\", \"nlmixr2\", \"Pharmpy\", \"Monolix\", \"exposure-response\", \"Emax\", \"EC50\", \"indirect response\", \"effect compartment\", \"TMDD\", \"PBPK\", \"bioequivalence\", \"RSABE\", \"ABEL\", \"allometric scaling\", \"first-in-human\", \"MABEL\", \"drug-drug interaction\", \"DDI\", \"ICH M12\", \"concentration-QTc\", \"therapeutic drug monitoring\", \"MIPD\", and \"dosing regimen\".","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpkpd-modeling\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M74Z86YHJJ1FM3X87BHN76KZ","pinecone","Guides use of Pinecone, a managed serverless vector database, through its Python client and the LangChain and LlamaIndex integrations. Covers creating indexes, upserting and querying vectors, metadata filtering, namespaces, hybrid dense and sparse search, index management, and deleting vectors. Use when building a production RAG system on a hosted vector store, adding semantic search or recommendations without running infrastructure, isolating per-user or per-tenant data with namespaces, combining dense and sparse vectors in one query, or filtering results by metadata. Do not use for self-hosted or local stores (Chroma, Weaviate) or offline similarity search (FAISS).","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpinecone\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":19},"01M74Z80T6E89SAJ8MYDD0767Q","pi-agent","Build with and use Pi, the minimal terminal coding harness. Use for installing Pi, configuring providers\u002Fmodels\u002Fsettings\u002Fenvironment variables, creating Pi skills\u002Fextensions\u002Fpackages\u002Fthemes\u002Fprompt templates, embedding Pi through the SDK, integrating over RPC or JSON event streams, parsing sessions, running local models through the llama.cpp router, developing custom Pi providers and TUI components, or using ecosystem packages such as pi-subagents (delegation\u002Forchestration), pi-mcp-adapter (MCP servers), pi-interview (interactive forms), and pi-web-access (web search, fetching, video understanding).","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpi-agent\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":19},"01M74Z7WF1Y8Q3VFAQPPPRAFMV","phoenix-observability","Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fphoenix-observability\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":19},"01M74Z7HNEH1A2AVYR3G1X2QV6","peft-fine-tuning","Fine-tunes LLMs with Hugging Face PEFT, using LoRA, QLoRA, IA3, AdaLoRA, prefix tuning, and prompt tuning so that under 1% of parameters are trained. Covers rank, alpha, and target module selection, loading and merging adapters, multi-adapter serving, and integration with TRL SFTTrainer, Axolotl, and vLLM. Use when fine-tuning 7B-70B models on limited GPU memory, when running QLoRA on a single 24GB GPU, when choosing LoRA rank and alpha, when merging or swapping adapters on one base model, or when debugging CUDA OOM or an adapter that does not apply. Not for full fine-tuning of small models under 1B parameters or cases needing all weights updated.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpeft-fine-tuning\u002FSKILL.md",[],[]]