Reject Summary Statistics

September 26, 2017

A summary statistic is what you get when you reduce a bunch of numbers down to a single number:

# the heights of 10 people you know (cm):
180 175 165 176 183 173 186 175 180 181

# the "average" of those heights:
177.4

But it doesn’t have to be the mean, it could be:

Wikipedia says:

[…] summary statistics are used to summarize a set of observations, in order to communicate the largest amount of information as simply as possible.

Why are summary statistics bad?

Summary statistics are extremely lossy.

They take all the data – full of nuances, patterns, outliers and special cases – and collapse it down to a single point.

What if the people you know included a basketball player?

# the heights of 10 people you know (cm):
180 175 165 176 183 173 198 175 180 181
#                       ^^^              ... changed 1 value

# the "average" of those heights:
178.6

Yes: 178.6 is bigger than 177.4 … but it doesn’t begin to tell the real story:

plot of heights

If you don’t look at the data, you don’t understand the data…

It’s almost a cliché, but Anscombe’s quartet clearly shows the limits of summary statistics:

Anscombe's quartet

Those four data sets have the same:

A more recent and striking example is the datasaurus dozen:

datasaurus data set

A terrible example: “average” request duration

If you’ve ever been asked:

What’s the average request duration?

for an HTTP endpoint, you know there’s only one good answer:

urgh…

The truth is that “it’s complicated” and the question itself is based on many wrong assumptions:

Here is a real duration graph distribution:

sample durations distribution

Details:

The mean duration sits exactly nowhere interesting or representative.

A failure of communication

Every time I hear/read the word “average”, I assume the worst. The average, or any summary statistics, obscures the real data, incompetently at best or maliciously at worst.

Summary statistics feel like information, but they are usually sound bites.

As a society, we need to strive for facts, for understanding and for objective data:

Discuss on Twitter