LLM metrics lab
================
Python 3.8+; standard library only; no network access or model weights needed.

1. Reproduce arithmetic and the pinned Qwen configuration audit:
   python -B metrics_lab.py
2. Reproduce the synthetic 100-question paired comparison:
   python -B paired_eval.py
3. Compare already-scored real results:
   python -B paired_eval.py --baseline baseline.jsonl --candidate candidate.jsonl --ids ids.json

Each JSONL row: {"id":"q001","correct":true,"status":"ok"}
The ID manifest is the complete expected list, e.g. ["q001"].
Status may be "ok" or "error"; an error must have correct=false and remains
in the denominator. Missing/extra/duplicate IDs are rejected.

These tools do not score text, train a model, or measure model quality.
Demo predictions are synthetic; Qwen values are a configuration/index audit.
Paired bootstrap assumes independent questions and two fixed checkpoints;
it does not account for clustered questions or training-seed variability.
--seed and --repeats control statistical resampling only. They do not train
checkpoints, generate new answers, or isolate decoding variability.

Article: https://chase6305.github.io/posts/ai/llm-training-metrics/
