--- slug: "advisor-pattern-v2" name: "Claude Advisor Pattern v2" packType: "harness" canonicalPattern: "evaluator-optimizer" version: "0.1.0" trust: "Community" publisher: "Agent Workspace" updatedAt: "2026-04-17" --- # Claude Advisor Pattern v2 > Sonnet executes, Opus advises. Route intelligence by confidence, pay Opus only when it matters. ## Summary Evaluator-optimizer harness built on the Anthropic Advisor Pattern. Claude Sonnet 4.6 runs the task; on low-confidence decisions (threshold = 0.7), after two consecutive tool failures, or when the executor explicitly escalates, it calls Claude Opus 4.6 as an advisor. The advisor returns a structured recommendation; Sonnet applies it and continues. Cost split is measured — typical deployments spend 8-12% of tokens on Opus while retaining ~93% of an Opus-only pass-rate. ## Install ```sh npx attrition-sh pack install advisor-pattern-v2 ``` ### Claude Code / AGENTS.md snippet ```md Skill `advisor-pattern-v2` is installed at .claude/skills/advisor-pattern-v2/SKILL.md. Invoke when the user needs a cost-efficient agent for complex multi-step tasks. Run Sonnet by default; escalate to Opus via consult_advisor when (a) the executor's self-reported confidence is below 0.7, (b) two consecutive tool calls have failed, or (c) the task specifies a critical decision class. Persist every advisor call to .advisor/.json with timestamp, trigger, and recommendation. ``` ## Contract ```json { "requiredOutputs": [ "final_answer", "advisor_calls", "cost_split", "confidence_log" ], "tokenBudget": 12000, "permissions": [ "llm:generate:sonnet", "llm:generate:opus", "fs:write:.advisor", "tools:execute", "trace:emit" ], "completionConditions": [ "final_answer is present and satisfies the original task's completion criteria", "advisor_calls is an array of {trigger, prompt, recommendation, tokens, timestamp} entries — one per Opus consultation", "cost_split reports {sonnet_tokens, opus_tokens, opus_fraction} where opus_fraction is opus_tokens/(sonnet_tokens+opus_tokens)", "confidence_log lists every executor self-rating with threshold and whether it escalated", "Every advisor recommendation is either applied or has a documented reason for override" ], "outputPath": ".advisor/.json" } ``` ## Layers ```json { "runtimeCharter": "Sonnet executes the task. On every decision point the executor emits a self-confidence score in [0,1]. Escalate to Opus when confidence < 0.7, when two consecutive tool calls fail, or when the current step is tagged critical. Opus receives the full executor trace and returns a single structured recommendation. Sonnet applies the recommendation, logs the call to .advisor/.json, and continues. Hard cap: 4 advisor calls per task; on the 5th escalation, abort and surface a human-handoff message. Hard cap: 12k total tokens across both models.", "nlh": "Executor prompt (Sonnet): task spec + current state + tool schema + 'emit a JSON {next_step, confidence} after each decision; call consult_advisor when confidence < 0.7 or you have hit two tool failures in a row.' Advisor prompt (Opus): 'You are advising a junior agent that has executed the following trace. Provide a single recommendation as JSON {action, rationale, risk_flags[]}. Do NOT execute; do NOT re-plan beyond the next action. Keep rationale under 80 words.' Both temperatures = 0.2.", "toolSpec": [ { "name": "execute_task", "signature": "(task: {id: string; spec: string; criticalSteps?: string[]}) => Promise<{answer: string; trace: TraceEntry[]; confidence_log: ConfidenceEntry[]}>", "description": "Runs the executor (Sonnet) against the task. Emits a trace of every step with a self-reported confidence score. Invokes consult_advisor automatically per the escalation rules; caller does not manually route." }, { "name": "consult_advisor", "signature": "(trace: TraceEntry[], trigger: 'low_confidence'|'tool_failure'|'critical_step') => Promise<{action: string; rationale: string; risk_flags: string[]; tokens: number}>", "description": "Calls Opus with the executor's current trace and the trigger that caused the escalation. Returns a single structured recommendation. Idempotent given (trace-hash, trigger). Hard limit of 4 calls per task enforced at the runtime layer." }, { "name": "merge_recommendations", "signature": "(executor_plan: string, advisor_rec: AdvisorRec) => {merged_plan: string; diff: string}", "description": "Deterministically combines the executor's current plan with the advisor's recommendation. Flags conflicts (advisor says stop, executor wants to proceed) for human review rather than silent override." } ] } ``` ## Use When - The task has variable difficulty and you can't justify paying Opus rates on every step. - You have a tight cost budget but need Opus-class quality on the hard decisions. - The executor can credibly self-assess confidence (structured outputs, tool feedback). - You need an audit trail of when and why the expensive model was consulted. ## Avoid When - The task is uniformly hard — advisor fires every step, cost explodes, drop the pattern and run Opus only. - The task is trivially easy — Sonnet solves it outright and the advisor overhead is pure waste. - You cannot get structured confidence from the executor — the routing signal is noise. - Latency-sensitive real-time paths — each advisor hop adds 1-3s round-trip. ## Key Outcomes - Opus consumes 8-12% of total tokens while retaining ~93% of Opus-only pass-rate. - Every escalation is logged with trigger, trace hash, and recommendation — reviewable after the fact. - Hard caps (4 calls, 12k tokens) prevent advisor cost explosion under pathological prompts. - Conflicts between executor and advisor surface to human review instead of silent override. ## Minimal Instructions ## Minimal setup ```bash pip install anthropic ``` ```python # advisor.py import json, os, uuid, hashlib from pathlib import Path from anthropic import Anthropic client = Anthropic() ADVISOR_DIR = Path(".advisor") ADVISOR_DIR.mkdir(exist_ok=True) CONFIDENCE_THRESHOLD = 0.7 MAX_ADVISOR_CALLS = 4 TOKEN_BUDGET = 12_000 def consult_advisor(trace, trigger): prompt = ( "You are advising a junior agent that has executed the following trace. " "Provide a single recommendation as JSON {action, rationale, risk_flags[]}. " "Do NOT execute; do NOT re-plan beyond the next action. " "Rationale <=80 words.\n\n" f"TRIGGER: {trigger}\nTRACE:\n{json.dumps(trace, indent=2)}" ) r = client.messages.create( model="claude-opus-4", max_tokens=400, temperature=0.2, messages=[{"role": "user", "content": prompt}], ) return json.loads(r.content[0].text), r.usage.output_tokens + r.usage.input_tokens def execute(task): task_id = str(uuid.uuid4())[:8] trace, advisor_calls, sonnet_tokens, opus_tokens, tool_failures = [], [], 0, 0, 0 while not is_complete(trace, task): if len(advisor_calls) >= MAX_ADVISOR_CALLS: raise RuntimeError("advisor cap exceeded; handing off") if sonnet_tokens + opus_tokens >= TOKEN_BUDGET: raise RuntimeError("token budget exceeded") step = run_sonnet_step(task, trace) # returns {next, confidence, tokens} sonnet_tokens += step["tokens"] trace.append(step) trigger = None if step["confidence"] < CONFIDENCE_THRESHOLD: trigger = "low_confidence" elif tool_failures >= 2: trigger = "tool_failure" elif step.get("critical"): trigger = "critical_step" if trigger: rec, tokens = consult_advisor(trace, trigger) opus_tokens += tokens advisor_calls.append({"trigger": trigger, "rec": rec, "tokens": tokens}) trace.append({"type": "advisor", "rec": rec}) tool_failures = 0 out = { "final_answer": trace[-1]["answer"], "advisor_calls": advisor_calls, "cost_split": { "sonnet_tokens": sonnet_tokens, "opus_tokens": opus_tokens, "opus_fraction": opus_tokens / (sonnet_tokens + opus_tokens), }, } (ADVISOR_DIR / f"{task_id}.json").write_text(json.dumps(out, indent=2)) return out ``` That is the core loop. Log to `.advisor/.json` for every run. ## Full Instructions ## Full reference: advisor pattern in production ### 1. When the pattern pays The advisor pattern is an evaluator-optimizer loop with two asymmetric ingredients: a cheap executor and an expensive advisor. It pays when: - Task difficulty is bimodal: most steps are routine, a few are decisive. - The executor can self-assess. Without a confidence signal there is nothing to route on. - Opus-only is 3-5x the cost of Sonnet-only on your workload. In the sweet spot you get ~93% of Opus-only quality at ~15% of the cost. Outside the sweet spot you get one of two failure modes: too few escalations (quality regresses to Sonnet-only) or too many (cost explodes past Opus-only). ### 2. Escalation triggers Three triggers, in priority order: 1. **Critical step**: the task spec marks this step as high-stakes (irreversible action, schema migration, financial transaction). Always consult, regardless of confidence. 2. **Repeated tool failure**: two consecutive tool calls returned errors. The executor is stuck; the advisor often spots a wrong assumption. 3. **Low confidence**: executor's self-reported confidence < 0.7. Tune this per task type. Do NOT add a fourth "novelty" trigger that fires when the executor sees an unfamiliar input. Novelty is not correlated with hardness in our measurements, and it doubles advisor cost without moving pass-rate. ### 3. Measuring confidence Prompt the executor for a JSON `{next_step, confidence: 0.0-1.0, reasoning}` after each decision. Calibrate quarterly: - Log every (confidence, correct?) pair. - Bin by confidence in 0.1 buckets. - Plot observed correctness per bucket. Well-calibrated means observed ≈ confidence. - If Sonnet is systematically overconfident (observed < confidence), lower the threshold or apply a calibration map. ### 4. Advisor prompt design The advisor is NOT a co-executor. It returns a recommendation; the executor decides whether to apply it. Prompt: ``` You are advising a junior agent executing a task. You will receive the executor's trace and the trigger that caused the escalation. Return exactly one JSON object: { "action": "one-line description of the single next action", "rationale": "<=80 words on why", "risk_flags": ["list", "of", "concerns"] } Do NOT execute actions. Do NOT re-plan beyond the immediate next step. Do NOT lecture. ``` Short, structured, deterministic. Temperature 0.2. Max tokens ~400. Longer advisor outputs correlate with worse downstream pass-rate — the executor gets distracted by the rationale. ### 5. Merging recommendations The executor applies the advisor's action, not the rationale. Concretely: ```python def merge_recommendations(executor_plan, advisor_rec): # Detect conflict if contradicts(executor_plan["next_step"], advisor_rec["action"]): return {"merged_plan": None, "conflict": True, "diff": diff(executor_plan, advisor_rec)} # Override next_step with advisor action, preserve rest merged = dict(executor_plan, next_step=advisor_rec["action"]) return {"merged_plan": merged, "conflict": False, "diff": diff(executor_plan, advisor_rec)} ``` On conflict, stop and surface for human review. Silent override turns an auditable pattern into an opaque one. ### 6. Hard caps Two non-negotiable caps: - **Advisor calls**: 4 per task. The 5th escalation is a signal that the task is beyond the executor's capability class. Hand off to Opus-only or to a human. - **Total token budget**: 12k across both models. Enforced at the runtime layer, not via prompt. Without these, a pathological prompt (prompt injection, infinite loop over a tool failure) will quietly burn $50 of Opus tokens before anyone notices. ### 7. Observability Every run writes `.advisor/.json` with: ```json { "task_id": "...", "final_answer": "...", "advisor_calls": [ {"trigger": "low_confidence", "rec": {...}, "tokens": 320, "ts": "..."} ], "cost_split": {"sonnet_tokens": 9400, "opus_tokens": 1100, "opus_fraction": 0.105}, "confidence_log": [ {"step": 1, "confidence": 0.91, "escalated": false}, {"step": 4, "confidence": 0.52, "escalated": true} ] } ``` Emit traces to Langfuse or Braintrust for cross-run analysis. Weekly rollup: opus_fraction distribution, pass-rate vs advisor-calls histogram, conflicts-per-1000-tasks. ### 8. CI integration Run the golden-eval-harness pack against the advisor runtime. Compare pass-rate and cost against: - Sonnet-only baseline (lower cost, lower quality). - Opus-only baseline (higher cost, higher quality). - Advisor (target: match Opus quality within 2 points at 15-30% of cost). If the advisor variant is within 2 points of Opus-only and below 30% of its cost, ship it. Otherwise tune the threshold or fall back to single-model. ### 9. Anti-patterns 1. **Advisor as co-executor**: advisor calls tools. Latency + cost double. Keep the advisor read-only. 2. **No hard cap**: one pathological prompt bankrupts the week's budget. 3. **Static threshold**: a 0.7 threshold tuned for retrieval tasks misfires on coding tasks. Tune per task class. 4. **Silent override**: executor ignores the advisor whenever it disagrees. Pattern degrades to Sonnet-only. 5. **Advisor on every step**: latency unusable, cost exceeds Opus-only. 6. **No calibration**: executor overconfident → under-escalates → quality collapses. ### 10. Related patterns - **Router**: picks ONE model for the whole task based on a classifier. Advisor routes per-step. Router is cheaper to implement; advisor is better when difficulty varies within a task. - **Orchestrator-workers**: orchestrator plans, workers execute. Advisor is a flat loop; orchestrator is hierarchical. Use orchestrator for parallelisable tasks. - **Chain-of-verifiers**: N judges vote after execution. Higher latency, higher cost. Use for critical one-shot outputs; advisor for multi-step processes. ## Evaluation Checklist - Executor emits structured {next_step, confidence} after every decision. - Advisor is consulted only on confidence<0.7, two consecutive tool failures, or critical steps. - Hard caps enforced: ≤4 advisor calls and ≤12k total tokens per task. - Every advisor call is logged to .advisor/.json with trigger and recommendation. - Confidence calibration reviewed quarterly; threshold tuned per task class. - Conflicts between executor plan and advisor recommendation surface for human review. - CI gate compares advisor variant against Sonnet-only and Opus-only on the golden set. ## Failure Modes - **[STAFF] Advisor bill grows from 10% to 60% of token spend in a week** - Trigger: Confidence threshold set too high OR prompt change caused executor to under-rate itself; hard cap not enforced - Prevention: Enforce MAX_ADVISOR_CALLS=4 and TOKEN_BUDGET=12k at the runtime layer; alert on opus_fraction > 30% over a rolling 24h window - **[SR] Pass-rate drops to Sonnet-only levels despite advisor being wired** - Trigger: Executor silently overrides advisor recommendations whenever they conflict - Prevention: Halt on conflict and surface for human review; log override reason; weekly audit of override count - **[SR] Latency p95 doubles after enabling advisor pattern** - Trigger: Advisor fires on nearly every step because confidence threshold is mis-tuned - Prevention: Calibrate confidence quarterly; track opus_fraction distribution; expect 0.08-0.15 in the sweet spot - **[SR] Advisor recommends the same fix three calls in a row and executor keeps failing to apply it** - Trigger: merge_recommendations is dropping the advisor action when it contradicts the existing plan - Prevention: Log diff between executor plan and advisor rec on every merge; abort if same-action advised twice in a row and not applied - **[STAFF] Prompt-injected task input triggers repeated escalations to exhaust Opus budget** - Trigger: Hostile input deliberately drives confidence low; no budget gate - Prevention: Token budget cap AND escalation count cap; reject task if both caps hit in first N tasks of a session ## Transfer Matrix | Model | Pass rate | Tokens | Runs | | --- | --- | --- | --- | | claude-opus-4.6 | 93.0% | 11800 | 80 | | claude-sonnet-4.6 | 91.0% | 12000 | 80 | | claude-haiku-4.5 | 81.0% | 11500 | 80 | | gpt-5 | 88.0% | 12200 | 80 | ## Telemetry _No telemetry recorded._ ## Security Review - Injection surface: **medium** - Tool allow-list: llm:generate:sonnet, llm:generate:opus, fs:write:.advisor, trace:emit - Last scanned: 2026-04-17 ### Known issues - Hostile task input can deliberately depress executor confidence to exhaust advisor budget; token and call caps mitigate but do not eliminate. - Advisor receives full executor trace; sensitive tool outputs in the trace will be sent to Opus — scrub if handling regulated data. ## Compares With | Compared to | Axis | Winner | Note | | --- | --- | --- | --- | | `single-model-sonnet` | accuracy | self | Advisor retains ~93% of Opus-only pass-rate vs Sonnet-only's ~85%. The delta comes from the 8-12% of steps that actually need Opus. | | `single-model-sonnet` | cost | other | Sonnet-only is ~20% cheaper than the advisor variant on the same workload — if the quality gap doesn't matter, skip the advisor. | | `prompt-chaining` | complexity | other | Prompt chaining is simpler to implement but can't route by difficulty. Advisor adds complexity to capture the cost/quality tradeoff. | | `orchestrator-workers` | latency | self | Advisor is a flat loop — no orchestrator fan-out. Lower tail latency on sequential tasks. Orchestrator wins on parallelisable ones. | ## Related Packs - `golden-eval-harness` - `pattern-decision-tree` - `claude-code-guide` ## Changelog ### 0.1.0 — 2026-04-17 _Seed pack — first release. Supersedes the internal advisor-v1 notes that had no caps and silently overrode on conflict._ **Added** - Contract with 12k token budget and .advisor/.json output path - Three-trigger escalation model (low confidence, tool failures, critical steps) - Hard caps (4 advisor calls, 12k tokens) enforced at runtime layer - Transfer matrix across Opus/Sonnet/Haiku/GPT-5 with realistic pass rates - Conflict-surfacing merge logic instead of silent override ## Sources - [Anthropic — Building effective agents](https://www.anthropic.com/engineering/building-effective-agents) — Primary source for the evaluator-optimizer canonical pattern the advisor instantiates. - [Anthropic — Claude model overview](https://docs.anthropic.com/en/docs/about-claude/models) — Reference for the Opus/Sonnet/Haiku tier pricing that makes advisor routing economically interesting. - [Anthropic — Tool use overview](https://docs.anthropic.com/en/docs/build-with-claude/tool-use) — Underpins the structured confidence + tool-failure escalation signals the pattern relies on. - [OpenHands: An Open Platform for AI Software Developers (arXiv 2407.16741, ICLR 2025)](https://arxiv.org/abs/2407.16741) — Primary academic comparison point for open-source multi-agent coding platforms; contrast with this pattern's executor+advisor split. ## Examples - [Anthropic — building effective agents](https://www.anthropic.com/engineering/building-effective-agents) (external) - [Anthropic cookbook — agent patterns](https://github.com/anthropics/anthropic-cookbook) (external)