Data Flywheel
The system loop that turns production traces into eval cases, fixes and training data under a review budget, a consent and minimum-tenant gate, a frozen holdout and a release-gate sequence, with what hosted Claude can and cannot train stated.
Last updated
After this section you can
- Route a production failure to the cheapest fix that works
- Gate production data for training with consent, scrub, a tenant cap and a frozen holdout
- Diagnose an inflated offline score as leakage before you retrain
24
Data Flywheel
Turn production failures into fixes you can prove: join each signal to its response, set the review budget, and gate the data before anything trains on it.
Key idea
Feedback is a test only once it is joined to the response that produced it. Each failure becomes an eval case or the cheapest fix; training comes last, and only what beats a frozen holdout ships.
Signals become fixes only through a join, a budget and a gate
Blue input, purple joined state, amber decisions, green output; teal is the loop. Pans sideways on a phone.