Answer Review and Quality Checks

Install

npx attrition-sh pack install answer-review-and-quality-checks

# Answer Review and Quality Checks
# See: /packs/answer-review-and-quality-checks

Summary

A runtime pattern that persists final answers, quality checks, evaluation metadata, and downstream review state as first-class records instead of burying them in message text.

Fit and expected payoff

When this pack earns its extra structure, when to skip it, and what it should improve.

Persist a final answer packet for every meaningful run.

The packet should include:
- final answer
- scope and references
- quality checks
- trace pointers
- evaluation linkage

Do not rely on chat text alone as the system of record.

Treat the answer as a durable application artifact.

For every non-trivial run:
1. Persist the final answer to an answer packet.
2. Attach references, quality checks, and trace metadata.
3. Link eval runs and live scoring back to the same packet shape.
4. Surface packet status in the UI so operators can see whether the answer passed or failed checks.

This should support:
- replay
- review
- later analytics
- quality gating before deployment

Evaluation checklist

These checks should pass before you consider the pattern production-ready.

• Is there a persisted answer packet for each completed assistant run?
• Do runtime checks and eval checks share a visible schema?
• Can a later reviewer inspect packet quality without reading the raw message transcript?

Common failure modes

Every check below traces back to a specific production failure. Read as: "I would think about X because in production Y can happen."

Official docs and implementation references

Reference implementations