Averages hide what your users feel

I’ve worked on product teams at startups and large companies, and one pattern keeps showing up. Users complain that the product is slow. Engineering pulls up a dashboard, shows the average response time, and the number looks fine. 200 milliseconds. Someone says “must be a one-off” or “probably their network” and the meeting moves on. Then the same complaints come back the next week, and the week after that, and nobody can figure out why because the metrics say the system is healthy.

Average latency tells you how the system performs in aggregate, and that sounds useful until you think about what it smooths over. If 95% of your requests come back in under 100 milliseconds and 5% take 3 seconds, the average still looks great. But nobody uses your product in aggregate. Every user gets one experience, and for a meaningful percentage of them, that experience is terrible. They don’t know they’re an outlier, they just know your product is slow.

Statistics is easy to get wrong, even for smart people. It was one of the more useful things I got out of school, and it shows up in engineering conversations more than you’d expect. The average is misleading for performance, because it hides the distribution behind a single comfortable number. And even the median, which a lot of teams use instead, has the same problem. It tells you what the typical user experiences, but it tells you nothing about the tail. Percentiles show the rest. P50 is the median. P95 is what the worst 1 in 20 experiences. P99 is the worst 1 in 100.

Think about it like an airport security line. Most people get through in five minutes. But every now and then someone triggers a bag search or sets off the scanner and they’re stuck for twenty minutes. The average wait time might be seven minutes, which sounds fine in a report. But if you’re the person behind the bag search, your experience is twenty minutes. Airports care about this, because if 5% of passengers wait twenty minutes, thousands of people miss flights.

Software is the same way. When we actually started looking at percentiles on a system I was working on, the picture looked nothing like the averages. Average: 200ms. P95: 1.2 seconds. P99: 8 seconds. Pmax: 25 seconds. The dashboard had been telling us everything was fine while some users were waiting nearly half a minute for a page to load. And when you do the math at scale, even “1 in 100” is not rare. If you’re handling 100,000 requests per minute, that’s 1,000 users every minute having a bad experience. That’s sixty thousand an hour, and most of those users won’t file a ticket. They’ll just leave.

And P95 is not enough. At real scale, even P99 starts to matter in absolute numbers. If you have a million requests a day and your P99 is 8 seconds, that’s 10,000 users a day staring at a screen wondering if your product is broken. And somewhere in there, someone is waiting 25 seconds. If you’re building something that millions of people use, you have to look at the full distribution, because the average and even the median can look fine while a real share of your users are having a terrible time.

This gets even worse in modern architectures where a single user request touches multiple services. Your API hits a database, checks a cache, calls an auth service, maybe talks to a third-party provider. Each of those has its own latency distribution, its own tail. If Service A has a P95 of 500 milliseconds and Service B also has a P95 of 500 milliseconds, the user’s experience is worse than 500 milliseconds, because the request needs both to finish, and the probability of hitting the tail of at least one goes up with every service in the chain. Google studied this problem extensively in “The Tail at Scale”, where fan-out across thousands of servers makes even a 0.01% tail event near-certain on every request. But you don’t need that kind of fan-out for this to matter. A B2B app with a few hundred users and three or four services in the chain will hit the same effect, just with smaller numbers that are easier to ignore and harder to prioritize.

None of this is new. Engineers who work on performance know about percentiles. But in the day-to-day, it’s easy to forget that the percentiles are the users. The dashboard shows a number, the number looks fine, and people move on. As a PM, one of the most useful things you can do on performance is push the conversation past the average. Ask what P95 and P99 look like, and what happens to the users at the edges.