reflexion-verbal-rl

/reflexion-verbal-rl

When to Use

• Agent failed a task and you want self-improvement without retraining
• Cross-session learning: storing failure reflections so next session avoids the same mistake
• Automated quality improvement over N trials (code gen, planning, reasoning tasks)
• Implementing CLAUDE.md-style "save memory" patterns with automatic failure analysis

Do NOT use for

• Tasks with no external evaluator and no ground truth (pure creative writing)
• Single-shot tasks where a second trial isn't possible
• Gradient-based fine-tuning workflows (Reflexion is prompt-only, no weight updates)

---

The three components + episodic memory

Actor          : generates a trajectory (any agent loop — ReAct, ReWOO, etc.)
Evaluator      : scores the trajectory → binary / heuristic / self-eval
Self-Reflector : writes natural-language reflection on the failure
Episodic memory: list of prior reflections, prepended to next trial's prompt

---

Reflexion loop (stdlib)

from dataclasses import dataclass, field
from typing import Callable

@dataclass
class ReflexionAgent:
    actor:      Callable[[str, list[str]], str]   # (task, reflections) -> trajectory
    evaluator:  Callable[[str, str], bool]         # (task, trajectory) -> success
    reflector:  Callable[[str, str], str]          # (task, trajectory) -> reflection
    max_trials: int = 5

    def run(self, task: str) -> tuple[str, list[str]]:
        reflections: list[str] = []

        for trial in range(self.max_trials):
            # Actor gets task + all prior reflections
            trajectory = self.actor(task, reflections)

            # Evaluator checks success
            if self.evaluator(task, trajectory):
                return trajectory, reflections

            # Self-Reflector diagnoses failure
            reflection = self.reflector(task, trajectory)
            reflections.append(f"[Trial {trial + 1}] {reflection}")
            print(f"Trial {trial + 1} failed. Reflection: {reflection}")

        return trajectory, reflections  # best effort after max_trials

---

Actor prompt (task + reflections)

def build_actor_prompt(task: str, reflections: list[str]) -> str:
    memory_block = ""
    if reflections:
        memory_block = "\n\nPast failure reflections (do NOT repeat these mistakes):\n"
        memory_block += "\n".join(f"- {r}" for r in reflections)

    return f"""Complete the following task.{memory_block}

Task: {task}

Respond with your reasoning and final answer."""

---

Three evaluator types

# 1. SCALAR — external binary (best signal)
def unit_test_evaluator(task: str, code: str) -> bool:
    import subprocess, tempfile, os
    with tempfile.NamedTemporaryFile(suffix=".py", delete=False, mode="w") as f:
        f.write(code)
        f.write("\n# Auto-test\nassert solution() is not None\n")
        fname = f.name
    result = subprocess.run(["python", fname], capture_output=True, timeout=10)
    os.unlink(fname)
    return result.returncode == 0

# 2. HEURISTIC — failure signature detection
def heuristic_evaluator(task: str, trajectory: str) -> bool:
    lines = trajectory.strip().split("\n")
    # Detect stuck loop: same action twice in a row
    actions = [l for l in lines if l.startswith("Action:")]
    if len(actions) >= 2 and actions[-1] == actions[-2]:
        return False
    # Detect trajectory too long
    if len(lines) > 50:
        return False
    return "ANSWER:" in trajectory

# 3. SELF-EVAL — LLM judges its own output (weakest; combine with heuristics)
def self_eval_prompt(task: str, trajectory: str) -> str:
    return f"""Did the agent successfully complete the task?

Task: {task}

Agent trajectory:
{trajectory}

Reply YES if complete and correct, NO if not. Start your reply with YES or NO."""

---

Self-Reflector prompt

def build_reflector_prompt(task: str, trajectory: str) -> str:
    return f"""You are analyzing a FAILED agent attempt.

Task: {task}

Failed trajectory:
{trajectory}

Write a concise reflection (2-3 sentences) identifying:
1. What went wrong
2. What the agent should do differently next time

Start with "I failed because..."."""

---

Where this pattern appears in 2026 production

CLAUDE.md / "save memory"   → failure captured as rule, prepended to next session
Letta sleep-time compute    → async agent reflects on past conversations → writes to memory blocks
YAMTAM /learn-rule          → corrections captured as explicit rules
LangGraph reflection node   → routes to refine sub-graph when evaluator fails
pro-workflow                → per-task reflections accumulate into agent CLAUDE.md

---

Anti-Fake-Pass Checklist

❌ History not passed to refiner → model repeats the same mistake on every trial
❌ Self-eval only (no external verifier) → LLM scores its own hallucination as correct
❌ No max_trials cap → infinite loop on tasks the model can never solve
❌ Reflection too vague ("try harder") → must name the specific mistake + specific fix
❌ Episodic memory grows unbounded → after 10+ trials, memory dominates context; add a trim
❌ Conflating Reflexion (verbal RL) with RLHF (gradient RL) → no weight updates here