[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BJMFXRHPZG78VYXGVMTY9J":3,"skill-same:01M7BJMFXRHPZG78VYXGVMTY9J":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":35,"commitSha":36,"skillDirectory":37,"forksCount":14,"repositoryHtmlUrl":38,"repositoryHomepageUrl":6,"ownerHtmlUrl":39,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":40,"install":41,"resources":54},"01M7BJMFXRHPZG78VYXGVMTY9J","mechanistic-interpretability-transformer-lens",null,"Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.","origin-labs-ai\u002Fcode","origin-labs-ai","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F347654492?v=4","01M7B0MAD1JMVZ6ME9891AYYYG","Origin","01M7B0MADP3AZS42R4NEF1JTS8",0,"MIT","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fpayloads\u002Fskills\u002Fmechanistic-interpretability-transformer-lens\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: mechanistic-interpretability-transformer-lens\ndescription: Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  author: Orchestra Research\n  tags: [Mechanistic Interpretability, TransformerLens, Activation Patching, Circuit Analysis]\n  dependencies: [transformer-lens>=2.0.0, torch>=2.0.0]\n---\n\n# TransformerLens: Mechanistic Interpretability for Transformers\n\nTransformerLens is the de facto standard library for mechanistic interpretability research on GPT-style language models. Created by Neel Nanda and maintained by Bryce Meyer, it provides clean interfaces to inspect and manipulate model internals via HookPoints on every activation.\n\n**GitHub**: [TransformerLensOrg\u002FTransformerLens](https:\u002F\u002Fgithub.com\u002FTransformerLensOrg\u002FTransformerLens) (2,900+ stars)\n\n## When to Use TransformerLens\n\n**Use TransformerLens when you need to:**\n- Reverse-engineer algorithms learned during training\n- Perform activation patching \u002F causal tracing experiments\n- Study attention patterns and information flow\n- Analyze circuits (e.g., induction heads, IOI circuit)\n- Cache and inspect intermediate activations\n- Apply direct logit attribution\n\n**Consider alternatives when:**\n- You need to work with non-transformer architectures → Use **nnsight** or **pyvene**\n- You want to train\u002Fanalyze Sparse Autoencoders → Use **SAELens**\n- You need remote execution on massive models → Use **nnsight** with NDIF\n- You want higher-level causal intervention abstractions → Use **pyvene**\n\n## Installation\n\n```bash\npip install transformer-lens\n```\n\nFor development version:\n```bash\npip install git+https:\u002F\u002Fgithub.com\u002FTransformerLensOrg\u002FTransformerLens\n```\n\n## Core Concepts\n\n### HookedTransformer\n\nThe main class that wraps transformer models with HookPoints on every activation:\n\n```python\nfrom transformer_lens import HookedTransformer\n\n# Load a model\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\n\n# For gated models (LLaMA, Mistral)\nimport os\nos.environ[\"HF_TOKEN\"] = \"your_token\"\nmodel = HookedTransformer.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n```\n\n### Supported Models (50+)\n\n| Family | Models |\n|--------|--------|\n| GPT-2 | gpt2, gpt2-medium, gpt2-large, gpt2-xl |\n| LLaMA | llama-7b, llama-13b, llama-2-7b, llama-2-13b |\n| EleutherAI | pythia-70m to pythia-12b, gpt-neo, gpt-j-6b |\n| Mistral | mistral-7b, mixtral-8x7b |\n| Others | phi, qwen, opt, gemma |\n\n### Activation Caching\n\nRun the model and cache all intermediate activations:\n\n```python\n# Get all activations\ntokens = model.to_tokens(\"The Eiffel Tower is in\")\nlogits, cache = model.run_with_cache(tokens)\n\n# Access specific activations\nresidual = cache[\"resid_post\", 5]  # Layer 5 residual stream\nattn_pattern = cache[\"pattern\", 3]  # Layer 3 attention pattern\nmlp_out = cache[\"mlp_out\", 7]  # Layer 7 MLP output\n\n# Filter which activations to cache (saves memory)\nlogits, cache = model.run_with_cache(\n    tokens,\n    names_filter=lambda name: \"resid_post\" in name\n)\n```\n\n### ActivationCache Keys\n\n| Key Pattern | Shape | Description |\n|-------------|-------|-------------|\n| `resid_pre, layer` | [batch, pos, d_model] | Residual before attention |\n| `resid_mid, layer` | [batch, pos, d_model] | Residual after attention |\n| `resid_post, layer` | [batch, pos, d_model] | Residual after MLP |\n| `attn_out, layer` | [batch, pos, d_model] | Attention output |\n| `mlp_out, layer` | [batch, pos, d_model] | MLP output |\n| `pattern, layer` | [batch, head, q_pos, k_pos] | Attention pattern (post-softmax) |\n| `q, layer` | [batch, pos, head, d_head] | Query vectors |\n| `k, layer` | [batch, pos, head, d_head] | Key vectors |\n| `v, layer` | [batch, pos, head, d_head] | Value vectors |\n\n## Workflow 1: Activation Patching (Causal Tracing)\n\nIdentify which activations causally affect model output by patching clean activations into corrupted runs.\n\n### Step-by-Step\n\n```python\nfrom transformer_lens import HookedTransformer, patching\nimport torch\n\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\n\n# 1. Define clean and corrupted prompts\nclean_prompt = \"The Eiffel Tower is in the city of\"\ncorrupted_prompt = \"The Colosseum is in the city of\"\n\nclean_tokens = model.to_tokens(clean_prompt)\ncorrupted_tokens = model.to_tokens(corrupted_prompt)\n\n# 2. Get clean activations\n_, clean_cache = model.run_with_cache(clean_tokens)\n\n# 3. Define metric (e.g., logit difference)\nparis_token = model.to_single_token(\" Paris\")\nrome_token = model.to_single_token(\" Rome\")\n\ndef metric(logits):\n    return logits[0, -1, paris_token] - logits[0, -1, rome_token]\n\n# 4. Patch each position and layer\nresults = torch.zeros(model.cfg.n_layers, clean_tokens.shape[1])\n\nfor layer in range(model.cfg.n_layers):\n    for pos in range(clean_tokens.shape[1]):\n        def patch_hook(activation, hook):\n            activation[0, pos] = clean_cache[hook.name][0, pos]\n            return activation\n\n        patched_logits = model.run_with_hooks(\n            corrupted_tokens,\n            fwd_hooks=[(f\"blocks.{layer}.hook_resid_post\", patch_hook)]\n        )\n        results[layer, pos] = metric(patched_logits)\n\n# 5. Visualize results (layer x position heatmap)\n```\n\n### Checklist\n- [ ] Define clean and corrupted inputs that differ minimally\n- [ ] Choose metric that captures behavior difference\n- [ ] Cache clean activations\n- [ ] Systematically patch each (layer, position) combination\n- [ ] Visualize results as heatmap\n- [ ] Identify causal hotspots\n\n## Workflow 2: Circuit Analysis (Indirect Object Identification)\n\nReplicate the IOI circuit discovery from \"Interpretability in the Wild\".\n\n### Step-by-Step\n\n```python\nfrom transformer_lens import HookedTransformer\nimport torch\n\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\n\n# IOI task: \"When John and Mary went to the store, Mary gave a bottle to\"\n# Model should predict \"John\" (indirect object)\n\nprompt = \"When John and Mary went to the store, Mary gave a bottle to\"\ntokens = model.to_tokens(prompt)\n\n# 1. Get baseline logits\nlogits, cache = model.run_with_cache(tokens)\n\njohn_token = model.to_single_token(\" John\")\nmary_token = model.to_single_token(\" Mary\")\n\n# 2. Compute logit difference (IO - S)\nlogit_diff = logits[0, -1, john_token] - logits[0, -1, mary_token]\nprint(f\"Logit difference: {logit_diff.item():.3f}\")\n\n# 3. Direct logit attribution by head\ndef get_head_contribution(layer, head):\n    # Project head output to logits\n    head_out = cache[\"z\", layer][0, :, head, :]  # [pos, d_head]\n    W_O = model.W_O[layer, head]  # [d_head, d_model]\n    W_U = model.W_U  # [d_model, vocab]\n\n    # Head contribution to logits at final position\n    contribution = head_out[-1] @ W_O @ W_U\n    return contribution[john_token] - contribution[mary_token]\n\n# 4. Map all heads\nhead_contributions = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)\nfor layer in range(model.cfg.n_layers):\n    for head in range(model.cfg.n_heads):\n        head_contributions[layer, head] = get_head_contribution(layer, head)\n\n# 5. Identify top contributing heads (name movers, backup name movers)\n```\n\n### Checklist\n- [ ] Set up task with clear IO\u002FS tokens\n- [ ] Compute baseline logit difference\n- [ ] Decompose by attention head contributions\n- [ ] Identify key circuit components (name movers, S-inhibition, induction)\n- [ ] Validate with ablation experiments\n\n## Workflow 3: Induction Head Detection\n\nFind induction heads that implement [A][B]...[A] → [B] pattern.\n\n```python\nfrom transformer_lens import HookedTransformer\nimport torch\n\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\n\n# Create repeated sequence: [A][B][A] should predict [B]\nrepeated_tokens = torch.tensor([[1000, 2000, 1000]])  # Arbitrary tokens\n\n_, cache = model.run_with_cache(repeated_tokens)\n\n# Induction heads attend from final [A] back to first [B]\n# Check attention from position 2 to position 1\ninduction_scores = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)\n\nfor layer in range(model.cfg.n_layers):\n    pattern = cache[\"pattern\", layer][0]  # [head, q_pos, k_pos]\n    # Attention from pos 2 to pos 1\n    induction_scores[layer] = pattern[:, 2, 1]\n\n# Heads with high scores are induction heads\ntop_heads = torch.topk(induction_scores.flatten(), k=5)\n```\n\n## Common Issues & Solutions\n\n### Issue: Hooks persist after debugging\n```python\n# WRONG: Old hooks remain active\nmodel.run_with_hooks(tokens, fwd_hooks=[...])  # Debug, add new hooks\nmodel.run_with_hooks(tokens, fwd_hooks=[...])  # Old hooks still there!\n\n# RIGHT: Always reset hooks\nmodel.reset_hooks()\nmodel.run_with_hooks(tokens, fwd_hooks=[...])\n```\n\n### Issue: Tokenization gotchas\n```python\n# WRONG: Assuming consistent tokenization\nmodel.to_tokens(\"Tim\")  # Single token\nmodel.to_tokens(\"Neel\")  # Becomes \"Ne\" + \"el\" (two tokens!)\n\n# RIGHT: Check tokenization explicitly\ntokens = model.to_tokens(\"Neel\", prepend_bos=False)\nprint(model.to_str_tokens(tokens))  # ['Ne', 'el']\n```\n\n### Issue: LayerNorm ignored in analysis\n```python\n# WRONG: Ignoring LayerNorm\npre_activation = residual @ model.W_in[layer]\n\n# RIGHT: Include LayerNorm\nln_scale = model.blocks[layer].ln2.w\nln_out = model.blocks[layer].ln2(residual)\npre_activation = ln_out @ model.W_in[layer]\n```\n\n### Issue: Memory explosion with large models\n```python\n# Use selective caching\nlogits, cache = model.run_with_cache(\n    tokens,\n    names_filter=lambda n: \"resid_post\" in n or \"pattern\" in n,\n    device=\"cpu\"  # Cache on CPU\n)\n```\n\n## Key Classes Reference\n\n| Class | Purpose |\n|-------|---------|\n| `HookedTransformer` | Main model wrapper with hooks |\n| `ActivationCache` | Dictionary-like cache of activations |\n| `HookedTransformerConfig` | Model configuration |\n| `FactoredMatrix` | Efficient factored matrix operations |\n\n## Integration with SAELens\n\nTransformerLens integrates with SAELens for Sparse Autoencoder analysis:\n\n```python\nfrom transformer_lens import HookedTransformer\nfrom sae_lens import SAE\n\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\nsae = SAE.from_pretrained(\"gpt2-small-res-jb\", \"blocks.8.hook_resid_pre\")\n\n# Run with SAE\ntokens = model.to_tokens(\"Hello world\")\n_, cache = model.run_with_cache(tokens)\nsae_acts = sae.encode(cache[\"resid_pre\", 8])\n```\n\n## Reference Documentation\n\nFor detailed API documentation, tutorials, and advanced usage, see the `references\u002F` folder:\n\n| File | Contents |\n|------|----------|\n| [references\u002FREADME.md](references\u002FREADME.md) | Overview and quick start guide |\n| [references\u002Fapi.md](references\u002Fapi.md) | Complete API reference for HookedTransformer, ActivationCache, HookPoints |\n| [references\u002Ftutorials.md](references\u002Ftutorials.md) | Step-by-step tutorials for activation patching, circuit analysis, logit lens |\n\n## External Resources\n\n### Tutorials\n- [Main Demo Notebook](https:\u002F\u002Ftransformerlensorg.github.io\u002FTransformerLens\u002Fgenerated\u002Fdemos\u002FMain_Demo.html)\n- [Activation Patching Demo](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FTransformerLensOrg\u002FTransformerLens\u002Fblob\u002Fmain\u002Fdemos\u002FActivation_Patching_in_TL_Demo.ipynb)\n- [ARENA Mech Interp Course](https:\u002F\u002Farena-foundation.github.io\u002FARENA\u002F) - 200+ hours of tutorials\n\n### Papers\n- [A Mathematical Framework for Transformer Circuits](https:\u002F\u002Ftransformer-circuits.pub\u002F2021\u002Fframework\u002Findex.html)\n- [In-context Learning and Induction Heads](https:\u002F\u002Ftransformer-circuits.pub\u002F2022\u002Fin-context-learning-and-induction-heads\u002Findex.html)\n- [Interpretability in the Wild (IOI)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2211.00593)\n\n### Official Documentation\n- [Official Docs](https:\u002F\u002Ftransformerlensorg.github.io\u002FTransformerLens\u002F)\n- [Model Properties Table](https:\u002F\u002Ftransformerlensorg.github.io\u002FTransformerLens\u002Fgenerated\u002Fmodel_properties_table.html)\n- [Neel Nanda's Glossary](https:\u002F\u002Fwww.neelnanda.io\u002Fmechanistic-interpretability\u002Fglossary)\n\n## Version Notes\n\n- **v2.0**: Removed HookedSAE (moved to SAELens)\n- **v3.0 (alpha)**: TransformerBridge for loading any nn.Module\n","# TransformerLens: Mechanistic Interpretability for Transformers\n\nTransformerLens is the de facto standard library for mechanistic interpretability research on GPT-style language models. Created by Neel Nanda and maintained by Bryce Meyer, it provides clean interfaces to inspect and manipulate model internals via HookPoints on every activation.\n\n**GitHub**: [TransformerLensOrg\u002FTransformerLens](https:\u002F\u002Fgithub.com\u002FTransformerLensOrg\u002FTransformerLens) (2,900+ stars)\n\n## When to Use TransformerLens\n\n**Use TransformerLens when you need to:**\n- Reverse-engineer algorithms learned during training\n- Perform activation patching \u002F causal tracing experiments\n- Study attention patterns and information flow\n- Analyze circuits (e.g., induction heads, IOI circuit)\n- Cache and inspect intermediate activations\n- Apply direct logit attribution\n\n**Consider alternatives when:**\n- You need to work with non-transformer architectures → Use **nnsight** or **pyvene**\n- You want to train\u002Fanalyze Sparse Autoencoders → Use **SAELens**\n- You need remote execution on massive models → Use **nnsight** with NDIF\n- You want higher-level causal intervention abstractions → Use **pyvene**\n\n## Installation\n\n```bash\npip install transformer-lens\n```\n\nFor development version:\n```bash\npip install git+https:\u002F\u002Fgithub.com\u002FTransformerLensOrg\u002FTransformerLens\n```\n\n## Core Concepts\n\n### HookedTransformer\n\nThe main class that wraps transformer models with HookPoints on every activation:\n\n```python\nfrom transformer_lens import HookedTransformer\n\n# Load a model\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\n\n# For gated models (LLaMA, Mistral)\nimport os\nos.environ[\"HF_TOKEN\"] = \"your_token\"\nmodel = HookedTransformer.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n```\n\n### Supported Models (50+)\n\n| Family | Models |\n|--------|--------|\n| GPT-2 | gpt2, gpt2-medium, gpt2-large, gpt2-xl |\n| LLaMA | llama-7b, llama-13b, llama-2-7b, llama-2-13b |\n| EleutherAI | pythia-70m to pythia-12b, gpt-neo, gpt-j-6b |\n| Mistral | mistral-7b, mixtral-8x7b |\n| Others | phi, qwen, opt, gemma |\n\n### Activation Caching\n\nRun the model and cache all intermediate activations:\n\n```python\n# Get all activations\ntokens = model.to_tokens(\"The Eiffel Tower is in\")\nlogits, cache = model.run_with_cache(tokens)\n\n# Access specific activations\nresidual = cache[\"resid_post\", 5]  # Layer 5 residual stream\nattn_pattern = cache[\"pattern\", 3]  # Layer 3 attention pattern\nmlp_out = cache[\"mlp_out\", 7]  # Layer 7 MLP output\n\n# Filter which activations to cache (saves memory)\nlogits, cache = model.run_with_cache(\n    tokens,\n    names_filter=lambda name: \"resid_post\" in name\n)\n```\n\n### ActivationCache Keys\n\n| Key Pattern | Shape | Description |\n|-------------|-------|-------------|\n| `resid_pre, layer` | [batch, pos, d_model] | Residual before attention |\n| `resid_mid, layer` | [batch, pos, d_model] | Residual after attention |\n| `resid_post, layer` | [batch, pos, d_model] | Residual after MLP |\n| `attn_out, layer` | [batch, pos, d_model] | Attention output |\n| `mlp_out, layer` | [batch, pos, d_model] | MLP output |\n| `pattern, layer` | [batch, head, q_pos, k_pos] | Attention pattern (post-softmax) |\n| `q, layer` | [batch, pos, head, d_head] | Query vectors |\n| `k, layer` | [batch, pos, head, d_head] | Key vectors |\n| `v, layer` | [batch, pos, head, d_head] | Value vectors |\n\n## Workflow 1: Activation Patching (Causal Tracing)\n\nIdentify which activations causally affect model output by patching clean activations into corrupted runs.\n\n### Step-by-Step\n\n```python\nfrom transformer_lens import HookedTransformer, patching\nimport torch\n\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\n\n# 1. Define clean and corrupted prompts\nclean_prompt = \"The Eiffel Tower is in the city of\"\ncorrupted_prompt = \"The Colosseum is in the city of\"\n\nclean_tokens = model.to_tokens(clean_prompt)\ncorrupted_tokens = model.to_tokens(corrupted_prompt)\n\n# 2. Get clean activations\n_, clean_cache = model.run_with_cache(clean_tokens)\n\n# 3. Define metric (e.g., logit difference)\nparis_token = model.to_single_token(\" Paris\")\nrome_token = model.to_single_token(\" Rome\")\n\ndef metric(logits):\n    return logits[0, -1, paris_token] - logits[0, -1, rome_token]\n\n# 4. Patch each position and layer\nresults = torch.zeros(model.cfg.n_layers, clean_tokens.shape[1])\n\nfor layer in range(model.cfg.n_layers):\n    for pos in range(clean_tokens.shape[1]):\n        def patch_hook(activation, hook):\n            activation[0, pos] = clean_cache[hook.name][0, pos]\n            return activation\n\n        patched_logits = model.run_with_hooks(\n            corrupted_tokens,\n            fwd_hooks=[(f\"blocks.{layer}.hook_resid_post\", patch_hook)]\n        )\n        results[layer, pos] = metric(patched_logits)\n\n# 5. Visualize results (layer x position heatmap)\n```\n\n### Checklist\n- [ ] Define clean and corrupted inputs that differ minimally\n- [ ] Choose metric that captures behavior difference\n- [ ] Cache clean activations\n- [ ] Systematically patch each (layer, position) combination\n- [ ] Visualize results as heatmap\n- [ ] Identify causal hotspots\n\n## Workflow 2: Circuit Analysis (Indirect Object Identification)\n\nReplicate the IOI circuit discovery from \"Interpretability in the Wild\".\n\n### Step-by-Step\n\n```python\nfrom transformer_lens import HookedTransformer\nimport torch\n\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\n\n# IOI task: \"When John and Mary went to the store, Mary gave a bottle to\"\n# Model should predict \"John\" (indirect object)\n\nprompt = \"When John and Mary went to the store, Mary gave a bottle to\"\ntokens = model.to_tokens(prompt)\n\n# 1. Get baseline logits\nlogits, cache = model.run_with_cache(tokens)\n\njohn_token = model.to_single_token(\" John\")\nmary_token = model.to_single_token(\" Mary\")\n\n# 2. Compute logit difference (IO - S)\nlogit_diff = logits[0, -1, john_token] - logits[0, -1, mary_token]\nprint(f\"Logit difference: {logit_diff.item():.3f}\")\n\n# 3. Direct logit attribution by head\ndef get_head_contribution(layer, head):\n    # Project head output to logits\n    head_out = cache[\"z\", layer][0, :, head, :]  # [pos, d_head]\n    W_O = model.W_O[layer, head]  # [d_head, d_model]\n    W_U = model.W_U  # [d_model, vocab]\n\n    # Head contribution to logits at final position\n    contribution = head_out[-1] @ W_O @ W_U\n    return contribution[john_token] - contribution[mary_token]\n\n# 4. Map all heads\nhead_contributions = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)\nfor layer in range(model.cfg.n_layers):\n    for head in range(model.cfg.n_heads):\n        head_contributions[layer, head] = get_head_contribution(layer, head)\n\n# 5. Identify top contributing heads (name movers, backup name movers)\n```\n\n### Checklist\n- [ ] Set up task with clear IO\u002FS tokens\n- [ ] Compute baseline logit difference\n- [ ] Decompose by attention head contributions\n- [ ] Identify key circuit components (name movers, S-inhibition, induction)\n- [ ] Validate with ablation experiments\n\n## Workflow 3: Induction Head Detection\n\nFind induction heads that implement [A][B]...[A] → [B] pattern.\n\n```python\nfrom transformer_lens import HookedTransformer\nimport torch\n\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\n\n# Create repeated sequence: [A][B][A] should predict [B]\nrepeated_tokens = torch.tensor([[1000, 2000, 1000]])  # Arbitrary tokens\n\n_, cache = model.run_with_cache(repeated_tokens)\n\n# Induction heads attend from final [A] back to first [B]\n# Check attention from position 2 to position 1\ninduction_scores = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)\n\nfor layer in range(model.cfg.n_layers):\n    pattern = cache[\"pattern\", layer][0]  # [head, q_pos, k_pos]\n    # Attention from pos 2 to pos 1\n    induction_scores[layer] = pattern[:, 2, 1]\n\n# Heads with high scores are induction heads\ntop_heads = torch.topk(induction_scores.flatten(), k=5)\n```\n\n## Common Issues & Solutions\n\n### Issue: Hooks persist after debugging\n```python\n# WRONG: Old hooks remain active\nmodel.run_with_hooks(tokens, fwd_hooks=[...])  # Debug, add new hooks\nmodel.run_with_hooks(tokens, fwd_hooks=[...])  # Old hooks still there!\n\n# RIGHT: Always reset hooks\nmodel.reset_hooks()\nmodel.run_with_hooks(tokens, fwd_hooks=[...])\n```\n\n### Issue: Tokenization gotchas\n```python\n# WRONG: Assuming consistent tokenization\nmodel.to_tokens(\"Tim\")  # Single token\nmodel.to_tokens(\"Neel\")  # Becomes \"Ne\" + \"el\" (two tokens!)\n\n# RIGHT: Check tokenization explicitly\ntokens = model.to_tokens(\"Neel\", prepend_bos=False)\nprint(model.to_str_tokens(tokens))  # ['Ne', 'el']\n```\n\n### Issue: LayerNorm ignored in analysis\n```python\n# WRONG: Ignoring LayerNorm\npre_activation = residual @ model.W_in[layer]\n\n# RIGHT: Include LayerNorm\nln_scale = model.blocks[layer].ln2.w\nln_out = model.blocks[layer].ln2(residual)\npre_activation = ln_out @ model.W_in[layer]\n```\n\n### Issue: Memory explosion with large models\n```python\n# Use selective caching\nlogits, cache = model.run_with_cache(\n    tokens,\n    names_filter=lambda n: \"resid_post\" in n or \"pattern\" in n,\n    device=\"cpu\"  # Cache on CPU\n)\n```\n\n## Key Classes Reference\n\n| Class | Purpose |\n|-------|---------|\n| `HookedTransformer` | Main model wrapper with hooks |\n| `ActivationCache` | Dictionary-like cache of activations |\n| `HookedTransformerConfig` | Model configuration |\n| `FactoredMatrix` | Efficient factored matrix operations |\n\n## Integration with SAELens\n\nTransformerLens integrates with SAELens for Sparse Autoencoder analysis:\n\n```python\nfrom transformer_lens import HookedTransformer\nfrom sae_lens import SAE\n\nmodel = HookedTransformer.from_pretrained(\"gpt2-small\")\nsae = SAE.from_pretrained(\"gpt2-small-res-jb\", \"blocks.8.hook_resid_pre\")\n\n# Run with SAE\ntokens = model.to_tokens(\"Hello world\")\n_, cache = model.run_with_cache(tokens)\nsae_acts = sae.encode(cache[\"resid_pre\", 8])\n```\n\n## Reference Documentation\n\nFor detailed API documentation, tutorials, and advanced usage, see the `references\u002F` folder:\n\n| File | Contents |\n|------|----------|\n| [references\u002FREADME.md](references\u002FREADME.md) | Overview and quick start guide |\n| [references\u002Fapi.md](references\u002Fapi.md) | Complete API reference for HookedTransformer, ActivationCache, HookPoints |\n| [references\u002Ftutorials.md](references\u002Ftutorials.md) | Step-by-step tutorials for activation patching, circuit analysis, logit lens |\n\n## External Resources\n\n### Tutorials\n- [Main Demo Notebook](https:\u002F\u002Ftransformerlensorg.github.io\u002FTransformerLens\u002Fgenerated\u002Fdemos\u002FMain_Demo.html)\n- [Activation Patching Demo](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FTransformerLensOrg\u002FTransformerLens\u002Fblob\u002Fmain\u002Fdemos\u002FActivation_Patching_in_TL_Demo.ipynb)\n- [ARENA Mech Interp Course](https:\u002F\u002Farena-foundation.github.io\u002FARENA\u002F) - 200+ hours of tutorials\n\n### Papers\n- [A Mathematical Framework for Transformer Circuits](https:\u002F\u002Ftransformer-circuits.pub\u002F2021\u002Fframework\u002Findex.html)\n- [In-context Learning and Induction Heads](https:\u002F\u002Ftransformer-circuits.pub\u002F2022\u002Fin-context-learning-and-induction-heads\u002Findex.html)\n- [Interpretability in the Wild (IOI)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2211.00593)\n\n### Official Documentation\n- [Official Docs](https:\u002F\u002Ftransformerlensorg.github.io\u002FTransformerLens\u002F)\n- [Model Properties Table](https:\u002F\u002Ftransformerlensorg.github.io\u002FTransformerLens\u002Fgenerated\u002Fmodel_properties_table.html)\n- [Neel Nanda's Glossary](https:\u002F\u002Fwww.neelnanda.io\u002Fmechanistic-interpretability\u002Fglossary)\n\n## Version Notes\n\n- **v2.0**: Removed HookedSAE (moved to SAELens)\n- **v3.0 (alpha)**: TransformerBridge for loading any nn.Module\n",{"name":5,"license":15,"metadata":24,"description":7},{"tags":25,"author":30,"version":31,"dependencies":32},[26,27,28,29],"Mechanistic Interpretability","TransformerLens","Activation Patching","Circuit Analysis","Orchestra Research","1.0.0",[33,34],"transformer-lens>=2.0.0","torch>=2.0.0","main","1e72c52e0f3ffd72fbc6ba994fda0f4a6d63911a","payloads\u002Fskills\u002Fmechanistic-interpretability-transformer-lens","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai","2026-11-14 20:35:25",{"source":8,"skillName":5,"commands":42,"prompt":52,"zipAvailable":53},[43,46,49],{"tool":44,"command":45},"npx","npx skills add origin-labs-ai\u002Fcode --skill mechanistic-interpretability-transformer-lens",{"tool":47,"command":48},"bunx","bunx skills add origin-labs-ai\u002Fcode --skill mechanistic-interpretability-transformer-lens",{"tool":50,"command":51},"pnpm","pnpm dlx skills add origin-labs-ai\u002Fcode --skill mechanistic-interpretability-transformer-lens","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fpayloads\u002Fskills\u002Fmechanistic-interpretability-transformer-lens\u002FSKILL.md",true,[55,61,65],{"path":56,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":59,"sourceHtmlUrl":60},"references\u002Fapi.md","REFERENCE","参考文档",8349,"https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fpayloads\u002Fskills\u002Fmechanistic-interpretability-transformer-lens\u002Freferences\u002Fapi.md",{"path":62,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002FREADME.md",1643,"https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fpayloads\u002Fskills\u002Fmechanistic-interpretability-transformer-lens\u002Freferences\u002FREADME.md",{"path":66,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Ftutorials.md",10142,"https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fpayloads\u002Fskills\u002Fmechanistic-interpretability-transformer-lens\u002Freferences\u002Ftutorials.md",[70,78,86,94,102,111],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":74,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":19},"01M7BP6Z7NBQXMB8Y5ZRR7PGCF","web-search","Implement web search capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to search for real-time information from the web, retrieve up-to-date content beyond the knowledge cutoff, or find the latest news and data. Returns structured search results with URLs, snippets, and metadata.","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fweb-search\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fweb-search\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":82,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M7BP6T13CWYHZSHGYG0RT885","web-reader","Implement web page content extraction capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to scrape web pages, extract article content, retrieve page metadata, or build applications that process web content. Supports automatic content extraction with title, HTML, and publication time retrieval.","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fweb-reader\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fweb-reader\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":90,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M7BP6GM9Z301PPT4ZRV049AN","video-understand","Implement specialized video understanding capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to analyze video content, understand motion and temporal sequences, extract information from video frames, describe video scenes, or perform video-based AI analysis. Optimized for MP4, AVI, MOV, and other common video formats.","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fvideo-understand\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fvideo-understand\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":98,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M7BP6ANRHQ1K7PVS11RBZCC7","Video Generation","Implement AI-powered video generation capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to generate videos from text prompts or images, create video content programmatically, or build applications that produce video outputs. Supports asynchronous task management with status polling and result retrieval.","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fvideo-generation\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fvideo-generation\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":106,"licenseFileUrl":107,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":19},"01M7BP56W5SX2MMJWKMJVC66F0","skill-creator","Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fskill-creator\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fskill-creator\u002FSKILL.md",[],[],{"skillNo":112,"name":113,"nameZh":6,"description":114,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":106,"licenseFileUrl":6,"sourceHtmlUrl":115,"categories":116,"keywords":117,"publishedAt":19},"01M7BP503HPG0DBCN4K0JKCA88","seo-content-writer","Use when the user asks to \"write SEO content\", \"create a blog post\", \"write an article\", \"content writing\", \"draft optimized content\", \"write me an article\", \"create a blog post about\", \"help me write SEO content\", or \"draft content for\". Creates high-quality, SEO-optimized content that ranks in search engines. Applies on-page SEO best practices, keyword optimization, and content structure for maximum visibility and engagement. For AI citation optimization, see geo-content-optimizer. For updating existing content, see content-refresher.","https:\u002F\u002Fgithub.com\u002Forigin-labs-ai\u002Fcode\u002Fblob\u002Fmain\u002Fwindows\u002Fsonar\u002Fmerger\u002Fcode-fusion-dead-20261101\u002Fskills\u002Fseo-content-writer\u002FSKILL.md",[],[]]