Python data script

# AGENTS.md: Python data scripts

## Folders
```
data/raw/        inputs exactly as received — never modified, often git-ignored
data/interim/    intermediate files written by scripts
output/          final CSVs, charts, reports
scripts/         one script per job, numbered if they run in order: 01_clean.py, 02_report.py
```

## Every script is runnable with one command
- Use PEP 723 inline metadata so `uv run scripts/01_clean.py` installs what it needs:
  ```python
  # /// script
  # requires-python = ">=3.12"
  # dependencies = ["polars>=1.0", "matplotlib"]
  # ///
  ```
- Add dependencies with `uv add --script scripts/01_clean.py <package>` rather than editing by hand.
- Paths come from CLI arguments (`argparse`) with sensible defaults, built with `pathlib.Path`. No hard-coded absolute paths.
- Put the work in `main()` and end with `if __name__ == "__main__": main()`.

## Data handling
- Prefer **polars** for new scripts (fast, strict types); use pandas if the project already does.
- Read with explicit types for IDs, postcodes and phone numbers (strings, so leading zeros survive) and explicit date parsing.
- Never overwrite files in `data/raw/`. Write outputs to new files.
- After each major step (load, filter, join, dedupe), log the row count. A join that changes the row count unexpectedly is a bug until proven otherwise.
- State assumptions in code comments next to the line that makes them ("drops refunds: amount < 0").
- Money: keep as integer pence/cents or `Decimal`, not floats, when sums must reconcile.
- Dates: be explicit about time zones and day-first formats (UK data is usually DD/MM/YYYY).

## Output
- Print a short summary at the end: rows in, rows out, files written.
- Charts: label axes with units, title says what the chart shows, save as PNG and SVG to `output/`.
- Do not print or commit personal data. If the data contains names, emails or addresses, keep outputs aggregated unless asked otherwise.

## Style
- Type hints on functions. Small functions per step (`load`, `clean`, `summarise`), each returning a dataframe.
- `ruff format` and `ruff check` clean (`uvx ruff check .`).

## Before you say "done"
Run the script end to end on the real input (or a sample), show the summary output, and mention any rows dropped and why.

# CLAUDE.md

@AGENTS.md

## Claude Code notes
- Skills from this loadout live in `.claude/skills/`: `data-sanity-check`. Use them when their description matches the task.
- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.

<!-- From the Agent Loadout "Python data script" v1.0.0 — https://loadout.jnw-it.com/loadouts/python-data-script/ -->

# GEMINI.md

@./AGENTS.md

## Gemini CLI notes
- Skills from this loadout live in `.agents/skills/`: `data-sanity-check`. Use them when their description matches the task.
- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.

<!-- From the Agent Loadout "Python data script" v1.0.0 — https://loadout.jnw-it.com/loadouts/python-data-script/ -->

---
description: Python data script conventions
globs: "**/*.py"
alwaysApply: false
---

- Scripts start with a PEP 723 `# /// script` block listing `requires-python` and `dependencies`; run them with `uv run`.
- `pathlib.Path` for paths, `argparse` for inputs, `main()` guarded by `if __name__ == "__main__":`.
- Never write into `data/raw/`.
- Log row counts after loads, filters, joins and dedupes.
- Read identifier-like columns (IDs, postcodes, phone numbers) as strings.
- No bare `except:`; catch specific exceptions and re-raise with context.
- Type hints on every function signature.

---
name: data-sanity-check
description: Profile a dataset (CSV, Excel, Parquet, JSON) and flag quality problems before analysis. Use when the user shares a data file, asks "is this data OK?", or before producing figures someone will rely on.
---

# Data sanity check

Write a throwaway script `scripts/00_sanity_check.py` (PEP 723, polars) that reads the file and prints the checks below.
Do not modify the input file.

## Checks
1. **Shape:** row and column count. Does it match what the human expects (ask if they know)?
2. **Types:** each column's inferred type. Flag numbers stored as text (e.g. "1,234", "£12.50"), dates stored as text,
   and IDs inferred as numbers (leading zeros lost).
3. **Nulls:** null count and percentage per column. Flag columns over 5% null and "null-like" strings: `""`, `"N/A"`, `"null"`, `"-"`, `"0"` in date columns.
4. **Duplicates:** fully duplicated rows, and duplicates on the column that should be a unique key.
5. **Ranges:** min/max for numbers and dates. Flag negatives where impossible, dates in the future or before 1990, outliers beyond 3 IQR.
6. **Categories:** for text columns with fewer than 50 distinct values, list the values with counts. Flag near-duplicates
   (`"London"`, `"london"`, `"London "`).
7. **Dates:** check whether day-first or month-first parsing is ambiguous (all days ≤ 12?). Note time zone assumptions.
8. **Joins (if a second file is involved):** key overlap in both directions, and whether the join is one-to-one,
   one-to-many or many-to-many. Many-to-many is almost always a mistake.
9. **Totals:** if there is an amount column, print the grand total so the human can reconcile it with the source system.
10. **Personal data:** flag columns that look like names, emails, phone numbers or addresses.

## Report
```
Sanity check — <file> — <rows> rows × <cols> columns
Problems
- [High] customer_id inferred as integer — 214 IDs had leading zeros. Read as string.
- [Medium] order_date: 3.1% null; 12 dates in 2031.
Looks fine
- No duplicate rows; amounts total £48,210.55.
Questions for you
- Is order_date day-first? Every day value is ≤ 12, so it's ambiguous.
```
Lead with problems, keep "looks fine" to a line or two, and end with questions only the human can answer.