{ "id": "evals-from-real-failures", "type": "loadout", "title": "Evals from real failures", "summary": "Build LLM and agent evals the practitioner way: read real traces first, tally failure categories, then turn each into code-checked regression cases.", "description": "Teams that ship LLM features keep giving the same advice: look at your data before writing an eval, and test for the failures you actually have rather than generic scores. This loadout makes that the default. The agent reads 30 to 50 real traces, notes the first failure in each, groups and counts the notes with a small script, and only then writes evals: code checks first, a binary model judge only where code cannot check, validated against human labels. Every case links to the trace it came from and must be shown to fail on the original bad output.\n", "highlights": [ "Error analysis before evals: one note per trace, categories after reading", "tally.py ranks failure categories with shares and example trace ids", "pytest regression runner with code checks; every case links to a real trace", "Model judges binary only, checked against human labels", "Each eval proven to fail on the original bad output" ], "version": "1.0.0", "license": "MIT", "author": { "name": "Agent Loadout", "url": "https://loadout.jnw-it.com" }, "credits": [ { "title": "Hamel Husain: LLM evals FAQ (error analysis first)", "url": "https://hamel.dev/blog/posts/evals-faq/" }, { "title": "@HamelHusain: eval skills that start from real traces", "url": "https://x.com/HamelHusain/status/2089438973714440196" }, { "title": "@Benogju: look at the data, start with around 30 traces", "url": "https://x.com/Benogju/status/2104930651573240043" }, { "title": "@pauliusztin_: replay your agent's real failures after every change", "url": "https://x.com/pauliusztin_/status/2109986591959626178" } ], "categories": [ "ai-agents", "data" ], "stacks": [ "python", "pytest", "uv" ], "files": [ { "path": "AGENTS.md", "role": "instructions", "description": "Order of work and rules for evals built from real traces.", "skill": null, "agent": null, "install_to": { "claude-code": "AGENTS.md", "codex": "AGENTS.md", "cursor": "AGENTS.md", "gemini-cli": "AGENTS.md", "antigravity-cli": "AGENTS.md", "copilot": "AGENTS.md" }, "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/AGENTS.md", "raw_latest_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/latest/AGENTS.md", "bytes": 1764, "lang": "markdown", "content": "# AGENTS.md: evals from real failures\n\nThis project has an LLM feature or agent. Evals here are built from what actually went wrong, not from a generic metric list.\n\n## Order of work\n1. Look at the data first. Before writing any eval, read real traces (inputs, tool calls, outputs). Start with 30 to 50.\n2. Error analysis: write a short free-text note on every failing trace, then group the notes into failure categories.\n Use the `error-analysis` skill. Count them; fix or evaluate the biggest categories first.\n3. Only then write evals, one per category that matters. Use the `failure-to-eval` skill.\n4. Every bug found in production becomes a regression case that links to its trace id.\n\n## Rules for evals\n- Prefer code checks (exact fields, JSON shape, forbidden strings, length, tool called or not) over model judges.\n- Model judges only for what code cannot check. Binary pass or fail with a one-line reason, never a 1 to 5 score.\n- Validate every judge against human labels before trusting it: report how often it agrees on passes and on fails separately.\n- No generic metrics (\"helpfulness\", \"coherence\") unless error analysis showed that failure.\n- Keep a held-out set. Never tune a prompt against the same cases you report.\n- An eval that has never failed tells you nothing. Show it fails on the original bad output before adding it.\n\n## Data handling\n- Traces can contain personal data. Work on exports in `evals/traces/` that are gitignored, and redact before committing a case.\n- Do not send traces to a third-party service the human has not approved.\n\n## Commands\n- Tally failure notes: `python3 /error-analysis/scripts/tally.py evals/notes.csv`\n- Run evals: `uv run --with pytest pytest evals -q` (or the project's own test command).\n" }, { "path": "CLAUDE.md", "role": "adapter", "description": "Imports AGENTS.md for Claude Code.", "skill": null, "agent": "claude-code", "install_to": { "claude-code": "CLAUDE.md", "codex": null, "cursor": null, "gemini-cli": null, "antigravity-cli": null, "copilot": null }, "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/CLAUDE.md", "raw_latest_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/latest/CLAUDE.md", "bytes": 402, "lang": "markdown", "content": "# CLAUDE.md\n\n@AGENTS.md\n\n## Claude Code notes\n- Skills from this loadout live in `.claude/skills/`: `error-analysis`, `failure-to-eval`. Use them when their description matches the task.\n- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.\n\n\n" }, { "path": "GEMINI.md", "role": "adapter", "description": "Imports AGENTS.md for Gemini CLI.", "skill": null, "agent": "gemini-cli", "install_to": { "claude-code": null, "codex": null, "cursor": null, "gemini-cli": "GEMINI.md", "antigravity-cli": null, "copilot": null }, "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/GEMINI.md", "raw_latest_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/latest/GEMINI.md", "bytes": 403, "lang": "markdown", "content": "# GEMINI.md\n\n@./AGENTS.md\n\n## Gemini CLI notes\n- Skills from this loadout live in `.agents/skills/`: `error-analysis`, `failure-to-eval`. Use them when their description matches the task.\n- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.\n\n\n" }, { "path": "skills/error-analysis/SKILL.md", "role": "skill", "description": "Read traces, note failures, group and count them.", "skill": "error-analysis", "agent": null, "install_to": { "claude-code": ".claude/skills/error-analysis/SKILL.md", "codex": ".agents/skills/error-analysis/SKILL.md", "cursor": ".agents/skills/error-analysis/SKILL.md", "gemini-cli": ".agents/skills/error-analysis/SKILL.md", "antigravity-cli": ".agents/skills/error-analysis/SKILL.md", "copilot": ".agents/skills/error-analysis/SKILL.md" }, "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/SKILL.md", "raw_latest_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/latest/skills/error-analysis/SKILL.md", "bytes": 1827, "lang": "markdown", "content": "---\nname: error-analysis\ndescription: Read real traces, note what went wrong in each, group notes into failure categories and count them. Use before writing any eval, after a model or prompt change, or when users report bad answers.\n---\n\n# Error analysis\n\nGoal: a ranked list of real failure modes, each with example trace ids.\n\n1. Export a sample of recent traces to `evals/traces/` (gitignored). Mix ordinary traffic with known complaints.\n 30 to 50 traces is enough to start; stop adding when new traces stop producing new kinds of failure.\n2. Create `evals/notes.csv` with the header `trace_id,pass,note,category`.\n3. Read each trace end to end: input, retrieved context, tool calls, final output. For each one write:\n - `pass`: `1` if a domain expert would accept the output as is, else `0`.\n - `note`: the first thing that went wrong, in plain words (\"ignored the date range\", \"called search twice\").\n Leave `category` empty for now. Do not invent categories before reading.\n4. Read all the notes together. Group them into 4 to 8 categories named after the failure, not the component\n (\"wrong date range\", not \"retrieval\"). Fill in `category` for every failing row.\n5. Tally: `python3 scripts/tally.py evals/notes.csv` prints each category with its count, share of failures and example\n trace ids, and lists failing rows that still have no category.\n6. Report the table to the human. Recommend: fix obvious bugs directly (a broken prompt line, a missing tool argument);\n write evals only for failures that will recur or are costly.\n\nRules:\n- One note per trace, about the first failure. Downstream errors usually follow from it.\n- Do not grade with a model at this stage. The point is for a person (or you, read carefully) to see the data.\n- Re-run this after every model or prompt change; categories move.\n" }, { "path": "skills/error-analysis/scripts/tally.py", "role": "reference", "description": "Ranks failure categories from a notes CSV.", "skill": "error-analysis", "agent": null, "install_to": { "claude-code": ".claude/skills/error-analysis/scripts/tally.py", "codex": ".agents/skills/error-analysis/scripts/tally.py", "cursor": ".agents/skills/error-analysis/scripts/tally.py", "gemini-cli": ".agents/skills/error-analysis/scripts/tally.py", "antigravity-cli": ".agents/skills/error-analysis/scripts/tally.py", "copilot": ".agents/skills/error-analysis/scripts/tally.py" }, "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/scripts/tally.py", "raw_latest_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/latest/skills/error-analysis/scripts/tally.py", "bytes": 1707, "lang": "markdown", "content": "#!/usr/bin/env python3\n\"\"\"Tally failure categories from an error-analysis notes CSV (trace_id,pass,note,category).\"\"\"\n\nimport argparse\nimport csv\nimport sys\nfrom collections import Counter, defaultdict\nfrom pathlib import Path\n\n\ndef main() -> int:\n ap = argparse.ArgumentParser(description=__doc__)\n ap.add_argument(\"notes\", type=Path)\n ap.add_argument(\"--examples\", type=int, default=3, help=\"example trace ids per category\")\n args = ap.parse_args()\n\n with args.notes.open(newline=\"\", encoding=\"utf-8\") as fh:\n rows = list(csv.DictReader(fh))\n missing = {\"trace_id\", \"pass\", \"note\", \"category\"} - set(rows[0].keys() if rows else [])\n if missing:\n print(f\"error: missing columns: {', '.join(sorted(missing))}\", file=sys.stderr)\n return 2\n\n failing = [r for r in rows if r[\"pass\"].strip() == \"0\"]\n counts = Counter(r[\"category\"].strip() for r in failing if r[\"category\"].strip())\n examples = defaultdict(list)\n for r in failing:\n cat = r[\"category\"].strip()\n if cat and len(examples[cat]) < args.examples:\n examples[cat].append(r[\"trace_id\"])\n uncategorised = [r[\"trace_id\"] for r in failing if not r[\"category\"].strip()]\n\n print(f\"traces: {len(rows)}, failing: {len(failing)} ({len(failing) / max(len(rows), 1):.0%})\")\n print(\"| category | count | share of failures | examples |\")\n print(\"| --- | --- | --- | --- |\")\n for cat, n in counts.most_common():\n print(f\"| {cat} | {n} | {n / len(failing):.0%} | {', '.join(examples[cat])} |\")\n if uncategorised:\n print(f\"uncategorised failures: {', '.join(uncategorised)}\")\n return 1\n return 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, { "path": "skills/failure-to-eval/SKILL.md", "role": "skill", "description": "Turn a failure category into regression cases.", "skill": "failure-to-eval", "agent": null, "install_to": { "claude-code": ".claude/skills/failure-to-eval/SKILL.md", "codex": ".agents/skills/failure-to-eval/SKILL.md", "cursor": ".agents/skills/failure-to-eval/SKILL.md", "gemini-cli": ".agents/skills/failure-to-eval/SKILL.md", "antigravity-cli": ".agents/skills/failure-to-eval/SKILL.md", "copilot": ".agents/skills/failure-to-eval/SKILL.md" }, "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/SKILL.md", "raw_latest_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/latest/skills/failure-to-eval/SKILL.md", "bytes": 1636, "lang": "markdown", "content": "---\nname: failure-to-eval\ndescription: Turn a failure category or a production bug into regression eval cases with code checks, and a model judge only when needed. Use after error analysis, or when a bad output is reported.\n---\n\n# Failure to eval\n\nInputs: a failure category from error analysis (or a reported bug) with real trace ids.\n\n1. For each trace, add one case to `evals/cases/.jsonl`:\n `{\"id\": \"...\", \"source\": \"\", \"input\": {...}, \"expect\": {...}}`. Redact personal data first.\n `source` is required: every case points at the real trace it came from.\n2. Write the expectation as code checks wherever possible. `references/test_regressions.py` supports:\n `contains`, `not_contains`, `max_chars`, `json_keys` (output must be JSON with these keys) and `matches` (regex).\n3. Copy `references/test_regressions.py` to `evals/` and set `EVAL_TARGET=module:function` to the function that produces\n the output (it receives the case `input` dict and returns a string).\n4. Prove the case bites: run it against the original bad output (or the old prompt) and confirm it fails. Then confirm it\n passes on the fix. A case that never failed is not a regression test.\n5. Only if code cannot express the check, write a judge prompt from `references/judge-prompt.md`: one failure mode per\n judge, binary pass or fail, a one-line reason. Label 20 or more outputs by hand, run the judge on them and report its\n agreement on passes and on fails separately. Do not use the judge until both are acceptable to the human.\n6. Run `uv run --with pytest pytest evals -q` and add the command to CI next to the unit tests.\n" }, { "path": "skills/failure-to-eval/references/test_regressions.py", "role": "reference", "description": "pytest runner for JSONL cases with code checks.", "skill": "failure-to-eval", "agent": null, "install_to": { "claude-code": ".claude/skills/failure-to-eval/references/test_regressions.py", "codex": ".agents/skills/failure-to-eval/references/test_regressions.py", "cursor": ".agents/skills/failure-to-eval/references/test_regressions.py", "gemini-cli": ".agents/skills/failure-to-eval/references/test_regressions.py", "antigravity-cli": ".agents/skills/failure-to-eval/references/test_regressions.py", "copilot": ".agents/skills/failure-to-eval/references/test_regressions.py" }, "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/test_regressions.py", "raw_latest_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/latest/skills/failure-to-eval/references/test_regressions.py", "bytes": 2381, "lang": "markdown", "content": "\"\"\"Regression evals built from real failures. Each case in evals/cases/*.jsonl links to the trace it came from.\n\nSet EVAL_TARGET=module:function. The function takes the case's input dict and returns the output string.\n\"\"\"\n\nimport importlib\nimport json\nimport os\nimport re\nfrom pathlib import Path\n\nimport pytest\n\nCASES_DIR = Path(os.environ.get(\"EVAL_CASES\", Path(__file__).parent / \"cases\"))\n\n\ndef load_cases():\n cases = []\n for path in sorted(CASES_DIR.glob(\"*.jsonl\")):\n for n, line in enumerate(path.read_text(encoding=\"utf-8\").splitlines(), 1):\n if line.strip():\n case = json.loads(line)\n case[\"_where\"] = f\"{path.name}:{n}\"\n cases.append(case)\n return cases\n\n\ndef target():\n module, _, func = os.environ[\"EVAL_TARGET\"].partition(\":\")\n return getattr(importlib.import_module(module), func)\n\n\ndef check(output: str, expect: dict) -> list[str]:\n problems = []\n for s in expect.get(\"contains\", []):\n if s not in output:\n problems.append(f\"missing {s!r}\")\n for s in expect.get(\"not_contains\", []):\n if s in output:\n problems.append(f\"contains forbidden {s!r}\")\n if \"max_chars\" in expect and len(output) > expect[\"max_chars\"]:\n problems.append(f\"{len(output)} chars > {expect['max_chars']}\")\n if \"matches\" in expect and not re.search(expect[\"matches\"], output):\n problems.append(f\"does not match /{expect['matches']}/\")\n if \"json_keys\" in expect:\n try:\n data = json.loads(output)\n absent = [k for k in expect[\"json_keys\"] if k not in data]\n if absent:\n problems.append(f\"JSON missing keys {absent}\")\n except (json.JSONDecodeError, TypeError):\n problems.append(\"output is not JSON\")\n return problems\n\n\nCASES = load_cases()\n\n\ndef test_every_case_links_to_a_real_trace():\n assert CASES, f\"no cases in {CASES_DIR}\"\n unlinked = [c[\"_where\"] for c in CASES if not str(c.get(\"source\", \"\")).strip()]\n assert not unlinked, f\"cases without a source trace id: {unlinked}\"\n\n\n@pytest.mark.parametrize(\"case\", CASES, ids=[c.get(\"id\", c[\"_where\"]) for c in CASES])\ndef test_case(case):\n output = target()(case[\"input\"])\n problems = check(output, case[\"expect\"])\n assert not problems, f\"{case['_where']} (trace {case.get('source')}): {'; '.join(problems)}\"\n" }, { "path": "skills/failure-to-eval/references/judge-prompt.md", "role": "reference", "description": "Binary judge template with a validation step.", "skill": "failure-to-eval", "agent": null, "install_to": { "claude-code": ".claude/skills/failure-to-eval/references/judge-prompt.md", "codex": ".agents/skills/failure-to-eval/references/judge-prompt.md", "cursor": ".agents/skills/failure-to-eval/references/judge-prompt.md", "gemini-cli": ".agents/skills/failure-to-eval/references/judge-prompt.md", "antigravity-cli": ".agents/skills/failure-to-eval/references/judge-prompt.md", "copilot": ".agents/skills/failure-to-eval/references/judge-prompt.md" }, "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/judge-prompt.md", "raw_latest_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/latest/skills/failure-to-eval/references/judge-prompt.md", "bytes": 596, "lang": "markdown", "content": "# Judge prompt template (one failure mode per judge)\n\nYou are checking one thing only: .\n\nPass if: .\nFail if: .\n\nInput:\n\n\nOutput to check:\n\n\nAnswer with JSON only: {\"pass\": true or false, \"reason\": \"\"}\n\nBefore use: label at least 20 outputs by hand, run this judge on them and report agreement on passes and on fails\nseparately. Rewrite the pass and fail lines, not the labels, until the human accepts both numbers.\n" } ], "requires": { "tools": [ "python3", "uv" ], "env": [], "mcp": [] }, "tested": { "status": "untested", "last_run": null, "fixture": null, "per_agent": { "claude-code": "untested", "codex": "untested", "cursor": "untested", "gemini-cli": "untested", "antigravity-cli": "untested", "copilot": "untested" } }, "security": { "scanned": true, "scanner": "intake-precheck/1", "verdict": "clean", "network": false, "scripts": [ "tally.py" ], "notes": "tally.py only reads a local CSV. The eval runner imports the function you name in EVAL_TARGET and runs it. The rules keep trace exports gitignored and off third-party services unless the human approves." }, "featured": false, "deprecated": false, "last_checked": "2026-10-17", "published": "2026-10-17", "updated": "2026-10-17", "compatibility": { "claude-code": "native", "codex": "native", "cursor": "native", "gemini-cli": "native", "antigravity-cli": "native", "copilot": "native" }, "counts": { "rules": 1, "skills": 2, "mcp": 0 }, "url": "https://loadout.jnw-it.com/loadouts/evals-from-real-failures/", "api_url": "https://loadout.jnw-it.com/api/loadouts/evals-from-real-failures.json", "verification": { "level": "verified-install", "status": "verified", "label": "Verified install", "verified": true, "checked_at": "2026-10-17T08:09:30Z", "commit": "03c1b9839c40eb1e44c0a2877baaa3978a96895c", "content_current": true, "agent_tests": "not-run", "counts": { "pass": 9, "fail": 0, "skip": 0, "not_applicable": 1 }, "meaning": "Level 1: automated checks on the files (parse, install paths, MCP config, safety scan, references, size, licence, starter build). No AI agent was run.", "checks": [ { "id": "listing", "name": "Listing", "status": "pass", "summary": "Schema valid; 8 files declared, none missing or extra", "details": [ "tested.status: untested (agent tests are separate from Level 1)" ] }, { "id": "parse", "name": "Files parse", "status": "pass", "summary": "All 8 files parse; front matter valid", "details": [] }, { "id": "adapters", "name": "Adapters", "status": "pass", "summary": "CLAUDE.md and GEMINI.md import AGENTS.md and are up to date", "details": [ "CLAUDE.md: matches the generator", "GEMINI.md: matches the generator" ] }, { "id": "install-paths", "name": "Install paths", "status": "pass", "summary": "Correct paths for all 6 agents; layouts re-discovered", "details": [ "Claude Code: 7 files, 2 skills in .claude/skills/", "Codex CLI: 6 files, 2 skills in .agents/skills/", "Cursor: 6 files, 2 skills in .agents/skills/", "Gemini CLI: 7 files, 2 skills in .agents/skills/", "Antigravity CLI: 6 files, 2 skills in .agents/skills/", "GitHub Copilot: 6 files, 2 skills in .agents/skills/" ] }, { "id": "mcp", "name": "MCP config", "status": "n/a", "summary": "No MCP servers in this loadout", "details": [] }, { "id": "safety", "name": "Safety scan", "status": "pass", "summary": "Clean: no credentials, risky shell or hidden text", "details": [ "Scanner intake-precheck/1: verdict clean; network no (listing says no)" ] }, { "id": "references", "name": "References", "status": "pass", "summary": "3 references between files resolve for every agent", "details": [] }, { "id": "size", "name": "Size limits", "status": "pass", "summary": "8 files, 10.5 KB total; largest skills/failure-to-eval/references/test_regressions.py 2.3 KB", "details": [ "Limits: 30 files, 32 KB per file, 128 KB total, 500 lines per skill or rule" ] }, { "id": "licence", "name": "Licence", "status": "pass", "summary": "MIT, by Agent Loadout; text in the repository LICENSE", "details": [] }, { "id": "starter-build", "name": "Starter build", "status": "pass", "summary": "tally.py ranks failure categories from a notes CSV and flags uncategorised failures; test_regressions.py fails on the original bad outputs and passes on the fix, and rejects cases without a source trace", "details": [ "Applied 13 loadout files at every agent's paths", "✓ tally failure categories: python3 .agents/skills/error-analysis/scripts/tally.py notes.csv", "✓ uncategorised failures are flagged: bash -c python3 .claude/skills/error-analysis/scripts/tally.py notes-incomplete.csv; test $? -eq 1 && echo flagged", "✓ copy the eval runner into evals/: cp .agents/skills/failure-to-eval/references/test_regressions.py evals/", "✓ evals fail on the original bad outputs: bash -c EVAL_TARGET=app:original PYTHONPATH=. uv run --no-project --with pytest==9.1.1 pytest evals -q -p no:cacheprovider; test $? -eq 1 && echo caught", "✓ evals pass on the fix: bash -c EVAL_TARGET=app:fixed PYTHONPATH=. uv run --no-project --with pytest==9.1.1 pytest evals -q -p no:cacheprovider", "✓ a case without a source trace is rejected: bash -c echo '{\"id\": \"x\", \"input\": {\"question\": \"q\", \"today\": \"2026-10-17\"}, \"expect\": {}}' >> evals/cases/internal-data.jsonl; EVAL_TARGET=app:fixed PYTHONPATH=. uv run --no-project --with pytest==9.1.1 pytest evals -q -p no:cacheprovider; test $? -eq 1 && echo rejected", "Covers: Failure categories are counted from failing traces only, with shares and example trace ids", "Covers: Failing rows without a category are listed and make the tally exit non-zero", "Covers: Code checks (contains, not_contains, max_chars, json_keys) catch the original failures and pass on the fix", "Covers: Every case must link to the real trace it came from" ] } ] }, "install": { "claude-code": { "files": [ { "path": "AGENTS.md", "target": "AGENTS.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/AGENTS.md", "role": "instructions" }, { "path": "CLAUDE.md", "target": "CLAUDE.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/CLAUDE.md", "role": "adapter" }, { "path": "skills/error-analysis/SKILL.md", "target": ".claude/skills/error-analysis/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/SKILL.md", "role": "skill" }, { "path": "skills/error-analysis/scripts/tally.py", "target": ".claude/skills/error-analysis/scripts/tally.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/scripts/tally.py", "role": "reference" }, { "path": "skills/failure-to-eval/SKILL.md", "target": ".claude/skills/failure-to-eval/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/SKILL.md", "role": "skill" }, { "path": "skills/failure-to-eval/references/test_regressions.py", "target": ".claude/skills/failure-to-eval/references/test_regressions.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/test_regressions.py", "role": "reference" }, { "path": "skills/failure-to-eval/references/judge-prompt.md", "target": ".claude/skills/failure-to-eval/references/judge-prompt.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/judge-prompt.md", "role": "reference" } ], "manual_steps": [], "zip_url": "https://loadout.jnw-it.com/download/evals-from-real-failures/claude-code.zip" }, "codex": { "files": [ { "path": "AGENTS.md", "target": "AGENTS.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/AGENTS.md", "role": "instructions" }, { "path": "skills/error-analysis/SKILL.md", "target": ".agents/skills/error-analysis/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/SKILL.md", "role": "skill" }, { "path": "skills/error-analysis/scripts/tally.py", "target": ".agents/skills/error-analysis/scripts/tally.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/scripts/tally.py", "role": "reference" }, { "path": "skills/failure-to-eval/SKILL.md", "target": ".agents/skills/failure-to-eval/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/SKILL.md", "role": "skill" }, { "path": "skills/failure-to-eval/references/test_regressions.py", "target": ".agents/skills/failure-to-eval/references/test_regressions.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/test_regressions.py", "role": "reference" }, { "path": "skills/failure-to-eval/references/judge-prompt.md", "target": ".agents/skills/failure-to-eval/references/judge-prompt.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/judge-prompt.md", "role": "reference" } ], "manual_steps": [], "zip_url": "https://loadout.jnw-it.com/download/evals-from-real-failures/codex.zip" }, "cursor": { "files": [ { "path": "AGENTS.md", "target": "AGENTS.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/AGENTS.md", "role": "instructions" }, { "path": "skills/error-analysis/SKILL.md", "target": ".agents/skills/error-analysis/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/SKILL.md", "role": "skill" }, { "path": "skills/error-analysis/scripts/tally.py", "target": ".agents/skills/error-analysis/scripts/tally.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/scripts/tally.py", "role": "reference" }, { "path": "skills/failure-to-eval/SKILL.md", "target": ".agents/skills/failure-to-eval/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/SKILL.md", "role": "skill" }, { "path": "skills/failure-to-eval/references/test_regressions.py", "target": ".agents/skills/failure-to-eval/references/test_regressions.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/test_regressions.py", "role": "reference" }, { "path": "skills/failure-to-eval/references/judge-prompt.md", "target": ".agents/skills/failure-to-eval/references/judge-prompt.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/judge-prompt.md", "role": "reference" } ], "manual_steps": [], "zip_url": "https://loadout.jnw-it.com/download/evals-from-real-failures/cursor.zip" }, "gemini-cli": { "files": [ { "path": "AGENTS.md", "target": "AGENTS.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/AGENTS.md", "role": "instructions" }, { "path": "GEMINI.md", "target": "GEMINI.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/GEMINI.md", "role": "adapter" }, { "path": "skills/error-analysis/SKILL.md", "target": ".agents/skills/error-analysis/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/SKILL.md", "role": "skill" }, { "path": "skills/error-analysis/scripts/tally.py", "target": ".agents/skills/error-analysis/scripts/tally.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/scripts/tally.py", "role": "reference" }, { "path": "skills/failure-to-eval/SKILL.md", "target": ".agents/skills/failure-to-eval/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/SKILL.md", "role": "skill" }, { "path": "skills/failure-to-eval/references/test_regressions.py", "target": ".agents/skills/failure-to-eval/references/test_regressions.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/test_regressions.py", "role": "reference" }, { "path": "skills/failure-to-eval/references/judge-prompt.md", "target": ".agents/skills/failure-to-eval/references/judge-prompt.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/judge-prompt.md", "role": "reference" } ], "manual_steps": [], "zip_url": "https://loadout.jnw-it.com/download/evals-from-real-failures/gemini-cli.zip" }, "antigravity-cli": { "files": [ { "path": "AGENTS.md", "target": "AGENTS.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/AGENTS.md", "role": "instructions" }, { "path": "skills/error-analysis/SKILL.md", "target": ".agents/skills/error-analysis/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/SKILL.md", "role": "skill" }, { "path": "skills/error-analysis/scripts/tally.py", "target": ".agents/skills/error-analysis/scripts/tally.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/scripts/tally.py", "role": "reference" }, { "path": "skills/failure-to-eval/SKILL.md", "target": ".agents/skills/failure-to-eval/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/SKILL.md", "role": "skill" }, { "path": "skills/failure-to-eval/references/test_regressions.py", "target": ".agents/skills/failure-to-eval/references/test_regressions.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/test_regressions.py", "role": "reference" }, { "path": "skills/failure-to-eval/references/judge-prompt.md", "target": ".agents/skills/failure-to-eval/references/judge-prompt.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/judge-prompt.md", "role": "reference" } ], "manual_steps": [], "zip_url": "https://loadout.jnw-it.com/download/evals-from-real-failures/antigravity-cli.zip" }, "copilot": { "files": [ { "path": "AGENTS.md", "target": "AGENTS.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/AGENTS.md", "role": "instructions" }, { "path": "skills/error-analysis/SKILL.md", "target": ".agents/skills/error-analysis/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/SKILL.md", "role": "skill" }, { "path": "skills/error-analysis/scripts/tally.py", "target": ".agents/skills/error-analysis/scripts/tally.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/error-analysis/scripts/tally.py", "role": "reference" }, { "path": "skills/failure-to-eval/SKILL.md", "target": ".agents/skills/failure-to-eval/SKILL.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/SKILL.md", "role": "skill" }, { "path": "skills/failure-to-eval/references/test_regressions.py", "target": ".agents/skills/failure-to-eval/references/test_regressions.py", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/test_regressions.py", "role": "reference" }, { "path": "skills/failure-to-eval/references/judge-prompt.md", "target": ".agents/skills/failure-to-eval/references/judge-prompt.md", "raw_url": "https://loadout.jnw-it.com/raw/evals-from-real-failures/1.0.0/skills/failure-to-eval/references/judge-prompt.md", "role": "reference" } ], "manual_steps": [], "zip_url": "https://loadout.jnw-it.com/download/evals-from-real-failures/copilot.zip" } }, "source_zip_url": "https://loadout.jnw-it.com/download/evals-from-real-failures/source.zip" }