Evals from real failures

# AGENTS.md: evals from real failures

This project has an LLM feature or agent. Evals here are built from what actually went wrong, not from a generic metric list.

## Order of work
1. Look at the data first. Before writing any eval, read real traces (inputs, tool calls, outputs). Start with 30 to 50.
2. Error analysis: write a short free-text note on every failing trace, then group the notes into failure categories.
   Use the `error-analysis` skill. Count them; fix or evaluate the biggest categories first.
3. Only then write evals, one per category that matters. Use the `failure-to-eval` skill.
4. Every bug found in production becomes a regression case that links to its trace id.

## Rules for evals
- Prefer code checks (exact fields, JSON shape, forbidden strings, length, tool called or not) over model judges.
- Model judges only for what code cannot check. Binary pass or fail with a one-line reason, never a 1 to 5 score.
- Validate every judge against human labels before trusting it: report how often it agrees on passes and on fails separately.
- No generic metrics ("helpfulness", "coherence") unless error analysis showed that failure.
- Keep a held-out set. Never tune a prompt against the same cases you report.
- An eval that has never failed tells you nothing. Show it fails on the original bad output before adding it.

## Data handling
- Traces can contain personal data. Work on exports in `evals/traces/` that are gitignored, and redact before committing a case.
- Do not send traces to a third-party service the human has not approved.

## Commands
- Tally failure notes: `python3 <skills dir>/error-analysis/scripts/tally.py evals/notes.csv`
- Run evals: `uv run --with pytest pytest evals -q` (or the project's own test command).

# CLAUDE.md

@AGENTS.md

## Claude Code notes
- Skills from this loadout live in `.claude/skills/`: `error-analysis`, `failure-to-eval`. Use them when their description matches the task.
- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.

<!-- From the Agent Loadout "Evals from real failures" v1.0.0 — https://loadout.jnw-it.com/loadouts/evals-from-real-failures/ -->

# GEMINI.md

@./AGENTS.md

## Gemini CLI notes
- Skills from this loadout live in `.agents/skills/`: `error-analysis`, `failure-to-eval`. Use them when their description matches the task.
- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.

<!-- From the Agent Loadout "Evals from real failures" v1.0.0 — https://loadout.jnw-it.com/loadouts/evals-from-real-failures/ -->

---
name: error-analysis
description: Read real traces, note what went wrong in each, group notes into failure categories and count them. Use before writing any eval, after a model or prompt change, or when users report bad answers.
---

# Error analysis

Goal: a ranked list of real failure modes, each with example trace ids.

1. Export a sample of recent traces to `evals/traces/` (gitignored). Mix ordinary traffic with known complaints.
   30 to 50 traces is enough to start; stop adding when new traces stop producing new kinds of failure.
2. Create `evals/notes.csv` with the header `trace_id,pass,note,category`.
3. Read each trace end to end: input, retrieved context, tool calls, final output. For each one write:
   - `pass`: `1` if a domain expert would accept the output as is, else `0`.
   - `note`: the first thing that went wrong, in plain words ("ignored the date range", "called search twice").
   Leave `category` empty for now. Do not invent categories before reading.
4. Read all the notes together. Group them into 4 to 8 categories named after the failure, not the component
   ("wrong date range", not "retrieval"). Fill in `category` for every failing row.
5. Tally: `python3 scripts/tally.py evals/notes.csv` prints each category with its count, share of failures and example
   trace ids, and lists failing rows that still have no category.
6. Report the table to the human. Recommend: fix obvious bugs directly (a broken prompt line, a missing tool argument);
   write evals only for failures that will recur or are costly.

Rules:
- One note per trace, about the first failure. Downstream errors usually follow from it.
- Do not grade with a model at this stage. The point is for a person (or you, read carefully) to see the data.
- Re-run this after every model or prompt change; categories move.

#!/usr/bin/env python3
"""Tally failure categories from an error-analysis notes CSV (trace_id,pass,note,category)."""

import argparse
import csv
import sys
from collections import Counter, defaultdict
from pathlib import Path


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("notes", type=Path)
    ap.add_argument("--examples", type=int, default=3, help="example trace ids per category")
    args = ap.parse_args()

    with args.notes.open(newline="", encoding="utf-8") as fh:
        rows = list(csv.DictReader(fh))
    missing = {"trace_id", "pass", "note", "category"} - set(rows[0].keys() if rows else [])
    if missing:
        print(f"error: missing columns: {', '.join(sorted(missing))}", file=sys.stderr)
        return 2

    failing = [r for r in rows if r["pass"].strip() == "0"]
    counts = Counter(r["category"].strip() for r in failing if r["category"].strip())
    examples = defaultdict(list)
    for r in failing:
        cat = r["category"].strip()
        if cat and len(examples[cat]) < args.examples:
            examples[cat].append(r["trace_id"])
    uncategorised = [r["trace_id"] for r in failing if not r["category"].strip()]

    print(f"traces: {len(rows)}, failing: {len(failing)} ({len(failing) / max(len(rows), 1):.0%})")
    print("| category | count | share of failures | examples |")
    print("| --- | --- | --- | --- |")
    for cat, n in counts.most_common():
        print(f"| {cat} | {n} | {n / len(failing):.0%} | {', '.join(examples[cat])} |")
    if uncategorised:
        print(f"uncategorised failures: {', '.join(uncategorised)}")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

---
name: failure-to-eval
description: Turn a failure category or a production bug into regression eval cases with code checks, and a model judge only when needed. Use after error analysis, or when a bad output is reported.
---

# Failure to eval

Inputs: a failure category from error analysis (or a reported bug) with real trace ids.

1. For each trace, add one case to `evals/cases/<category>.jsonl`:
   `{"id": "...", "source": "<trace id>", "input": {...}, "expect": {...}}`. Redact personal data first.
   `source` is required: every case points at the real trace it came from.
2. Write the expectation as code checks wherever possible. `references/test_regressions.py` supports:
   `contains`, `not_contains`, `max_chars`, `json_keys` (output must be JSON with these keys) and `matches` (regex).
3. Copy `references/test_regressions.py` to `evals/` and set `EVAL_TARGET=module:function` to the function that produces
   the output (it receives the case `input` dict and returns a string).
4. Prove the case bites: run it against the original bad output (or the old prompt) and confirm it fails. Then confirm it
   passes on the fix. A case that never failed is not a regression test.
5. Only if code cannot express the check, write a judge prompt from `references/judge-prompt.md`: one failure mode per
   judge, binary pass or fail, a one-line reason. Label 20 or more outputs by hand, run the judge on them and report its
   agreement on passes and on fails separately. Do not use the judge until both are acceptable to the human.
6. Run `uv run --with pytest pytest evals -q` and add the command to CI next to the unit tests.

"""Regression evals built from real failures. Each case in evals/cases/*.jsonl links to the trace it came from.

Set EVAL_TARGET=module:function. The function takes the case's input dict and returns the output string.
"""

import importlib
import json
import os
import re
from pathlib import Path

import pytest

CASES_DIR = Path(os.environ.get("EVAL_CASES", Path(__file__).parent / "cases"))


def load_cases():
    cases = []
    for path in sorted(CASES_DIR.glob("*.jsonl")):
        for n, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
            if line.strip():
                case = json.loads(line)
                case["_where"] = f"{path.name}:{n}"
                cases.append(case)
    return cases


def target():
    module, _, func = os.environ["EVAL_TARGET"].partition(":")
    return getattr(importlib.import_module(module), func)


def check(output: str, expect: dict) -> list[str]:
    problems = []
    for s in expect.get("contains", []):
        if s not in output:
            problems.append(f"missing {s!r}")
    for s in expect.get("not_contains", []):
        if s in output:
            problems.append(f"contains forbidden {s!r}")
    if "max_chars" in expect and len(output) > expect["max_chars"]:
        problems.append(f"{len(output)} chars > {expect['max_chars']}")
    if "matches" in expect and not re.search(expect["matches"], output):
        problems.append(f"does not match /{expect['matches']}/")
    if "json_keys" in expect:
        try:
            data = json.loads(output)
            absent = [k for k in expect["json_keys"] if k not in data]
            if absent:
                problems.append(f"JSON missing keys {absent}")
        except (json.JSONDecodeError, TypeError):
            problems.append("output is not JSON")
    return problems


CASES = load_cases()


def test_every_case_links_to_a_real_trace():
    assert CASES, f"no cases in {CASES_DIR}"
    unlinked = [c["_where"] for c in CASES if not str(c.get("source", "")).strip()]
    assert not unlinked, f"cases without a source trace id: {unlinked}"


@pytest.mark.parametrize("case", CASES, ids=[c.get("id", c["_where"]) for c in CASES])
def test_case(case):
    output = target()(case["input"])
    problems = check(output, case["expect"])
    assert not problems, f"{case['_where']} (trace {case.get('source')}): {'; '.join(problems)}"

# Judge prompt template (one failure mode per judge)

You are checking one thing only: <failure mode, in the words from error analysis>.

Pass if: <concrete description of acceptable output>.
Fail if: <concrete description of the failure, with one real example>.

Input:
<input>

Output to check:
<output>

Answer with JSON only: {"pass": true or false, "reason": "<one sentence>"}

Before use: label at least 20 outputs by hand, run this judge on them and report agreement on passes and on fails
separately. Rewrite the pass and fail lines, not the labels, until the human accepts both numbers.