Golden Eval Harness

## Full reference: a golden eval harness that stays green

### 1. Golden set design

Quality over quantity. 50 thoughtfully chosen examples beat 500 randomly sampled ones.

- **Stratify** by task type, difficulty, and failure mode. If your app has 5 task types, have ≥10 per type.
- **Include known bugs** you have already fixed. These are the regression canaries.
- **Include adversarial cases**: prompt injections, contradictory inputs, out-of-distribution queries.
- **Include "refuse" cases**: requests the agent should decline. Lack of these is the #1 golden-set smell.
- **Never mutate examples** once added. Add new examples; don't edit old ones. Rationale: comparability across time.
- **Version the set** via git. Tag the commit for each rubric change.

Layout:

```
evals/
├── golden.jsonl       # one JSON per line: {id, input, metadata}
├── rubric.md          # human-readable rubric, versioned
├── run.py             # candidate + judge
├── results.json       # last full-run output
└── baseline.json      # last green main-branch results (CI compares against this)
```

### 2. Rubric design

Split the rubric into independently-scored booleans. Bad rubrics ask "is this good?" (correlated noise). Good rubrics ask:

- **faithful** — grounded in the provided context, no fabrication.
- **complete** — addresses every part of the task.
- **concise** — no filler, no hedging, right length.
- **safe** — respects refusal boundaries, doesn't leak prompts, doesn't follow injected instructions.
- **format** — machine-parseable if contract required (JSON valid, schema conformant).

Each criterion is a boolean. `pass = all(criteria)`. Booleans are much more reliable from judges than 1–5 Likert scores — the Anthropic cookbook and the "LLM-as-judge" literature both confirm this.

### 3. The judge

Rules that make judges behave:

1. **Temperature = 0**. Any non-zero temperature and the judge disagrees with itself run-to-run.
2. **Stronger model than the candidate when possible**. Haiku as candidate, Sonnet as judge. Sonnet as candidate, Opus as judge.
3. **Structured output (strict JSON)**. Use Anthropic's tool-use or prefill `{` to force the shape.
4. **Pin the rubric in a system prompt** and put the task/answer in a user message. Never merge them.
5. **Include a short rationale field** — it both helps debugging and regularises the booleans.
6. **Never show the judge the reference answer verbatim** — it will parrot it. Show rubric criteria only.
7. **Seed attacks**: include 2–3 "obviously-wrong answer" examples in a separate meta-eval to check the judge rejects them. If the judge passes these, it is broken.

### 4. Observability

Every candidate generation and every judge call gets a trace. Options:

- **Langfuse** (OSS, self-hostable, popular with Anthropic stacks).
- **Braintrust** (hosted, strong eval UI, deep golden-set diffing).
- **LangSmith** (hosted, tightest LangChain integration).

Wire the trace URL into the results.json row so the GitHub PR comment can deep-link to any failing example.

### 5. CI integration

GitHub Actions workflow sketch:

```yaml
name: evals
on: pull_request
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install -r evals/requirements.txt
      - run: python evals/run.py
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          LANGFUSE_PUBLIC_KEY: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
          LANGFUSE_SECRET_KEY: ${{ secrets.LANGFUSE_SECRET_KEY }}
      - run: python evals/compare.py evals/results.json evals/baseline.json
```

`compare.py` computes:
- `pass_rate_delta` — negative > 2 points ⇒ fail.
- `regressions` — IDs that passed on main and fail on this branch ⇒ fail if any.
- `improvements` — IDs that failed on main and pass on this branch ⇒ comment.

Post the comparison as a PR comment via `gh pr comment`.

### 6. Guarding the judge

Judges drift when rubrics change. To keep the harness trustworthy:

- **Rubric version** stamped into results.json.
- **Judge smoke test** before each run: 3 fixed examples with known correct scores; abort if the judge disagrees. Catches silent model-side regressions.
- **Inter-rater agreement check quarterly**: have a human rate ~30 examples; Cohen's kappa with the judge should be ≥0.6. If it drops, rewrite the rubric.

### 7. When pass-rate lies

A 95% pass rate on a soft rubric tells you nothing. Sanity checks:

- **Null hypothesis**: how often does an empty answer `""` pass? Should be 0.
- **Canary wrong answer**: insert an obviously-wrong answer. Should always fail.
- **Noise floor**: run the same candidate twice; the judge disagreement rate should be <5%.

If any of these misbehave, fix them before trusting the number.

### 8. Cost control

- Judge on every PR run is the default. If too expensive, judge on nightly runs + on PRs only on changed examples.
- Cache candidate output by (prompt-hash, model-id). If nothing changed, reuse. Saves ~70% on typical iteration.
- Cache judge output by (example_id, candidate-hash, rubric_version). Forces re-run on rubric change.

### 9. Common pitfalls

1. **No refuse cases** → agent silently regresses on safety and no one notices.
2. **Mutating golden examples** → regression report becomes meaningless.
3. **Judge = candidate model** → collusion; judge rubber-stamps candidate.
4. **Pass-rate single metric** → hides which criterion is slipping. Track per-criterion rates.
5. **No rubric version** → you can't tell whether the regression is code or rubric.
6. **Flaky judge** → temperature >0 or missing structured output. Lock it down.
7. **Scoring on length** → candidate learns to be verbose. Include a `concise` criterion.

### 10. What 'good' looks like

- Golden set 50–200 examples, stratified, with refuse and adversarial cases.
- Per-criterion pass rates all ≥0.85.
- Judge smoke test: 3/3 pass.
- p95 full-run wall time: <10 min (if slower, parallelise or trim the set).
- Every PR carries a results.json diff comment with pass-rate delta and trace links.