Statistics: How Many Would You Check?

April 9, 2015

Russian Translation, thanks to Vlad.
Ukrainian Translation, thanks to Sandi.
Estonian Translation, thanks to Johanne.
Spanish Translation, thanks to Laura.
French Translation, thanks to Jean-Etienne.
Urdu Translation, thanks to Samuel.
Sindhi Translation, thanks to Samuel.

Imagine this situation:

You just performed a batch update on millions of users in your database. There were no error messages and you are confident that everything went well. But it wouldn’t hurt to check…

How many users would you have to check to feel confident that everything worked for at least 95% of users?

Here are some thoughts:

I don’t think the answer is obvious. I had to take some time to think about it.

A detour: average rating

I remembered reading How Not To Sort By Average Rating, and I thought I could apply the same logic to this problem.

If you only have one review, and it’s positive, is that 100%? Intuitively, we know that it’s not: it’s just one person’s opinion. As more and more people give positive reviews, we can start feeling better about the accuracy of the score.

The quote from the article is:

Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what?

We can use the lower bound of the Wilson confidence interval.

In practice, with R

The binom.wilson function, from the binom package, can be used like this:

> binom.wilson(18, 20)
    method  x  n mean     lower     upper
  1 wilson 18 20  0.9 0.6989664 0.9721335

In other words, if we sampled 18 positives and 2 negatives (18/20), the “real” fraction probably falls between 0.699 and 0.972 (mean: 0.9).

For our example, we could invoke it with 100% success:

> binom.wilson(10, 10)
    method  x  n mean     lower upper
  1 wilson 10 10    1 0.7224672     1

The upper bound isn’t interesting: we’re not interested in the best case scenario. But if you check 10 and they are all successful, you can feel confident that it worked for (lower bound) 72.2% of users.

If we keep checking, and we keep finding successes, we can feel more and more certain about “true” success:

lower bound success over the number of checks

It takes 73 checks to reach a lower bound of 95% of “true” success (the red line).

Analysis

Here is the analysis as a RMarkdown document and the resulting output html document.

Discuss on Twitter