1:1 mentoring with Big Tech AI engineers
System Design

The First 24 Hours

What the pager cannot see on launch day, the three alarms that can see it, the provider limits you will meet on the first ramp, the one rollback lever to type in advance, and which cost binds at your mix.

Last updated

Production9 min readFirst readRelease Pipeline

After this section you can

  • Arm three alarms on emitted metrics, each with a minimum-volume guard and a zero-traffic check
  • Name the provider limits that bite on a launch ramp, and the one 429 you never retry
  • Choose the rollback lever by blast radius, and say which cost binds at a given mix
31

The First 24 Hours

Three alarms that see what a pager misses, the launch limits, and one rollback lever typed in advance.

Key idea

The pager stays quiet for the failures that cost the most. Alarm on what users and invoices feel, with a minimum volume; learn the launch limits before the ramp; type the rollback lever in advance.

Three alarms and one lever, armed before the first real user
THREE ALARMS, EACH ON A COUNTER Errors 5xx ratio over 2% at least 50 requests in 2 hours req_total Spend Hourly cost over $2 at least 20 runs an hour cost_total Quality shadow Max-steps over 5%, or escalations over 25% at least 30 runs an hour One rollback lever, typed before launch Flag (seconds), prompt pointer (seconds), digest (minutes) a page sends a person here

Red: alarms, each on a counter with a guard. Green: the one lever. Thresholds are examples. Pans sideways on a phone.

Related

More in System Design

Get full access to all 74+ sections with code examples, diagrams, and interactive animations.

Unlock Premium