Claude Advisor Pattern v2

## Full reference: advisor pattern in production

### 1. When the pattern pays

The advisor pattern is an evaluator-optimizer loop with two asymmetric ingredients: a cheap executor and an expensive advisor. It pays when:

- Task difficulty is bimodal: most steps are routine, a few are decisive.
- The executor can self-assess. Without a confidence signal there is nothing to route on.
- Opus-only is 3-5x the cost of Sonnet-only on your workload.

In the sweet spot you get ~93% of Opus-only quality at ~15% of the cost. Outside the sweet spot you get one of two failure modes: too few escalations (quality regresses to Sonnet-only) or too many (cost explodes past Opus-only).

### 2. Escalation triggers

Three triggers, in priority order:

1. **Critical step**: the task spec marks this step as high-stakes (irreversible action, schema migration, financial transaction). Always consult, regardless of confidence.
2. **Repeated tool failure**: two consecutive tool calls returned errors. The executor is stuck; the advisor often spots a wrong assumption.
3. **Low confidence**: executor's self-reported confidence < 0.7. Tune this per task type.

Do NOT add a fourth "novelty" trigger that fires when the executor sees an unfamiliar input. Novelty is not correlated with hardness in our measurements, and it doubles advisor cost without moving pass-rate.

### 3. Measuring confidence

Prompt the executor for a JSON `{next_step, confidence: 0.0-1.0, reasoning}` after each decision. Calibrate quarterly:

- Log every (confidence, correct?) pair.
- Bin by confidence in 0.1 buckets.
- Plot observed correctness per bucket. Well-calibrated means observed ≈ confidence.
- If Sonnet is systematically overconfident (observed < confidence), lower the threshold or apply a calibration map.

### 4. Advisor prompt design

The advisor is NOT a co-executor. It returns a recommendation; the executor decides whether to apply it. Prompt:

```
You are advising a junior agent executing a task. You will receive the
executor's trace and the trigger that caused the escalation. Return exactly
one JSON object:

{
  "action": "one-line description of the single next action",
  "rationale": "<=80 words on why",
  "risk_flags": ["list", "of", "concerns"]
}

Do NOT execute actions. Do NOT re-plan beyond the immediate next step.
Do NOT lecture.
```

Short, structured, deterministic. Temperature 0.2. Max tokens ~400. Longer advisor outputs correlate with worse downstream pass-rate — the executor gets distracted by the rationale.

### 5. Merging recommendations

The executor applies the advisor's action, not the rationale. Concretely:

```python
def merge_recommendations(executor_plan, advisor_rec):
    # Detect conflict
    if contradicts(executor_plan["next_step"], advisor_rec["action"]):
        return {"merged_plan": None, "conflict": True, "diff": diff(executor_plan, advisor_rec)}
    # Override next_step with advisor action, preserve rest
    merged = dict(executor_plan, next_step=advisor_rec["action"])
    return {"merged_plan": merged, "conflict": False, "diff": diff(executor_plan, advisor_rec)}
```

On conflict, stop and surface for human review. Silent override turns an auditable pattern into an opaque one.

### 6. Hard caps

Two non-negotiable caps:

- **Advisor calls**: 4 per task. The 5th escalation is a signal that the task is beyond the executor's capability class. Hand off to Opus-only or to a human.
- **Total token budget**: 12k across both models. Enforced at the runtime layer, not via prompt.

Without these, a pathological prompt (prompt injection, infinite loop over a tool failure) will quietly burn $50 of Opus tokens before anyone notices.

### 7. Observability

Every run writes `.advisor/<task-id>.json` with:

```json
{
  "task_id": "...",
  "final_answer": "...",
  "advisor_calls": [
    {"trigger": "low_confidence", "rec": {...}, "tokens": 320, "ts": "..."}
  ],
  "cost_split": {"sonnet_tokens": 9400, "opus_tokens": 1100, "opus_fraction": 0.105},
  "confidence_log": [
    {"step": 1, "confidence": 0.91, "escalated": false},
    {"step": 4, "confidence": 0.52, "escalated": true}
  ]
}
```

Emit traces to Langfuse or Braintrust for cross-run analysis. Weekly rollup: opus_fraction distribution, pass-rate vs advisor-calls histogram, conflicts-per-1000-tasks.

### 8. CI integration

Run the golden-eval-harness pack against the advisor runtime. Compare pass-rate and cost against:

- Sonnet-only baseline (lower cost, lower quality).
- Opus-only baseline (higher cost, higher quality).
- Advisor (target: match Opus quality within 2 points at 15-30% of cost).

If the advisor variant is within 2 points of Opus-only and below 30% of its cost, ship it. Otherwise tune the threshold or fall back to single-model.

### 9. Anti-patterns

1. **Advisor as co-executor**: advisor calls tools. Latency + cost double. Keep the advisor read-only.
2. **No hard cap**: one pathological prompt bankrupts the week's budget.
3. **Static threshold**: a 0.7 threshold tuned for retrieval tasks misfires on coding tasks. Tune per task class.
4. **Silent override**: executor ignores the advisor whenever it disagrees. Pattern degrades to Sonnet-only.
5. **Advisor on every step**: latency unusable, cost exceeds Opus-only.
6. **No calibration**: executor overconfident → under-escalates → quality collapses.

### 10. Related patterns

- **Router**: picks ONE model for the whole task based on a classifier. Advisor routes per-step. Router is cheaper to implement; advisor is better when difficulty varies within a task.
- **Orchestrator-workers**: orchestrator plans, workers execute. Advisor is a flat loop; orchestrator is hierarchical. Use orchestrator for parallelisable tasks.
- **Chain-of-verifiers**: N judges vote after execution. Higher latency, higher cost. Use for critical one-shot outputs; advisor for multi-step processes.