AGENT SKILL

needle-haystack-context

暂无简介

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 13
Forks
2

03 SKILL.md 正文

/needle-haystack-context

When to Use

  • Test whether an LLM can retrieve a specific fact buried in a long context window
  • Choose optimal placement for critical information in agent system prompts
  • Diagnose why an agent "forgets" instructions in long conversations
  • Benchmark model context quality before deploying for long-context RAG tasks

Do NOT use for

  • Short contexts < 4k tokens (all models handle these reliably)
  • Semantic search (use [[in-memory-vector-storage]] for retrieval)

Lost-in-the-Middle effect

Research finding (Liu et al. 2023):
  Models perform best when critical info is at START or END of context.
  Performance degrades ~40% when critical info is buried in the MIDDLE.

  0%──────────────────────100% position in context
  ████████░░░░░░░░░░░░████████  ← retrieval accuracy
  Best    Worst     Best

Agent implication: put rules/constraints at the TOP of system prompt,
  not buried in the middle of a long document list.

Needle-in-Haystack test runner

import anthropic

def needle_test(
  client:    anthropic.Anthropic,
  needle:    str,   # fact to retrieve: "The magic number is 42"
  haystack:  str,   # long filler text (Paul Graham essays, etc.)
  position:  float, # 0.0 = start, 0.5 = middle, 1.0 = end
  ctx_len:   int,   # target context length in tokens
) -> bool:
  # Place needle at the specified position
  words   = haystack.split()
  insert  = int(len(words) * position)
  padded  = ' '.join(words[:insert]) + f'\n\n{needle}\n\n' + ' '.join(words[insert:])
  trimmed = padded[:ctx_len * 4]  # ~4 chars/token

  response = client.messages.create(
    model      = 'claude-sonnet-4-6',
    max_tokens = 64,
    system     = 'Answer based only on the provided text.',
    messages   = [
      { 'role': 'user', 'content': f'{trimmed}\n\nWhat is the magic number?' }
    ],
  )
  answer = response.content[0].text
  return '42' in answer

Sweep test across positions and context lengths

import numpy as np

positions = [0.0, 0.1, 0.25, 0.5, 0.75, 0.9, 1.0]
ctx_lens  = [2048, 4096, 8192, 16384]
results   = {}

for ctx in ctx_lens:
  for pos in positions:
    hit = needle_test(client, needle, haystack, pos, ctx)
    results[(ctx, pos)] = hit
    print(f'ctx={ctx:6d} pos={pos:.1f} → {"PASS" if hit else "FAIL"}')

# Identify weak spots
failures = [(ctx, pos) for (ctx, pos), ok in results.items() if not ok]
print('\nFailed positions:', failures)

Agent prompt ordering rules (from NIAH findings)

Priority order for system prompt sections:
  1. [TOP]    Hard constraints, safety rules, tool restrictions
  2. [TOP]    Agent identity and role
  3. [MIDDLE] Reference documents, context (expendable)
  4. [BOTTOM] Specific task instructions
  5. [BOTTOM] Output format requirements

NEVER bury "do not do X" rules in the middle of a long document list.
If context > 32k tokens: summarize middle sections, keep rules at edges.

Anti-Fake-Pass Checklist

❌ Testing with model's advertised context length but not checking quality → model "accepts" 128k but retrieves poorly past 32k
❌ Needle is too similar to haystack content → false positives (model guesses correctly without retrieval)
❌ Single position test → must sweep all positions to find the degradation curve
❌ Counting tokens by character/4 for needle placement → use tiktoken for accurate position
❌ Using this test for semantic understanding → NIAH tests literal retrieval only
❌ Fixing NIAH failures by just shortening context → also restructure prompt (put key info at top)

04 同仓库更多 Skills

Horonomos-laps
Horonomos-laps/Yana-AI

openai--sentry--sentry

Use when the user asks to inspect Sentry issues or events, summarize recent production errors, or pull basic Sentry health data via the Sentry API; perform read-only queries with the bundled script and require `SENTRY_AUTH_TOKEN`.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

ito-basket-compare

Compare Itô prediction-market baskets against a user's knowledge base, portfolio notes, financial context, watchlist, or research thesis. Use for read-only basket comparison and gap analysis without investment advice or live trading.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

implementing-network-traffic-baselining

Build network traffic baselines from NetFlow/IPFIX data using Python pandas for statistical analysis, z-score anomaly detection, and hourly/daily traffic pattern profiling

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

detecting-evasion-techniques-in-endpoint-logs

Detects defense evasion techniques used by adversaries in endpoint logs including log tampering, timestomping, process injection, and security tool disabling. Use when investigating suspicious endpoint behavior, building detection rules for evasion tactics, or conducting threat hunting for stealthy adversary activity. Activates for requests involving evasion detection, defense evasion analysis, log tampering detection, or MITRE ATT&CK TA0005.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

implementing-microsegmentation-with-guardicore

Implementing microsegmentation using Akamai Guardicore Segmentation to map application dependencies, create granular network policies, visualize east-west traffic flows, and enforce least-privilege communication between workloads across data centers and cloud.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

ramdisk-tmpfs-patterns

暂无简介

★ 13