A practical system for giving every request a time limit, catching regressions in review and keeping p99 honest as the product grows.
Every product team has a number they quietly avoid. For us it was p99 latency on the dashboard, a figure that hovered around 2.4 seconds for most of last year while the median looked perfectly healthy. Users did not complain about the median. They complained about Monday mornings.
This is the story of how we gave every request a budget, made that budget visible in code review, and brought the slowest one percent of page loads under 600 milliseconds without a rewrite.