Confidence intervals for creatives

As a data scientist working for an advertising agency a huge part of the job is running A/B tests. It could be an A/B test on creative, audiences, platforms or placements. Thankfully, I work in an agency with a huge focus on testing and learning with a goal of taking a lot of the subjectivity out of what is produced. This allows for a lot of tests to be run which as a data scientist is great, however, it means you come across statistical significance very often and it can be a challenge to explain that sometimes a 5% difference in the click-through rate of two ads does not mean we can say with confidence one ad is better than the other. Here I will attempt to explain statistical significance and confidence intervals to those with a very little maths/stats background. I have titled it confidence intervals for creatives, purely for the alteration and it sounded fun. Many creatives will understand everything I am about to talk about.

For this explanation let’s deal with click-through rate (CTR) which is the rate at which people who have seen a digital ad and have clicked on it.

Let’s say that ad A had a CTR of 15% and ad B had a CTR of 10% and that as a result we are 90% confidence that ad A will perform better than ad B. Let’s explain.

First, we need to understand confidence intervals and for that let’s play some darts. Imagine you throw a dart at the dartboard and you hit the bullseye. Right bang in the centre (this is important). For the purpose of the explanation, let’s say that you throwing the dart is the experiment (or letting the ad out into the digital world) and the bullseye is the recorded CTR. Now imagine you throw the dart again! How confident are you that you would get the exact point you previously hit. Unless you are an exceptional darts player probably not very confident. How about hitting somewhere within the first ring, the triple ring? I bit more confident I imagine. And the second ring, the double ring? Even more so. This is the idea of a confidence interval. If you reran the experiment, put the ad out to the world again or threw the dart, how confident you are in getting a range around the initial hit/value.

Obviously, the confidence interval of the experiment doesn’t depend on how good you are at darts. In fact, it depends on the number of participants in the experiment or in the case of our ad, the number of people who have seen it. The more people who have seen it, the narrower the range of values you are confident the CTR would be if you put the ad out there again. i.e the area around the bullseye.

Now back to the initial statement. We will say that ad A had a CTR of 15% and ad B had a CTR of 10% and we are 90% confidence with the results. Here we have essentially run two separate experiments, one for ad A and the other for ad B. So in our dartboard analogy, we have thrown two at two different dartboards. One where the bullseye is 10% and the other where the bullseye is 15%. Now, let’s say we have a good amount of people view the ad, which if you recall is equivalent to how good our dart player is. So we are 90% confident that if our dart player threw again, he/she would get within the double ring. If you lined up the two dartboards on a line, where the centres were located at 10% and 15%, and the double rings do not overlap. Then you can say that ad A performs better than ad B with 90% confidence.

A long-winded explanation perhaps. Thankfully, we can mathematically calculate confidence intervals based on participant size. A final word on interpreting the results. Saying ad A will perform better than ad B with statistical significance is the same as saying ad A will perform better than ad B with 95% confidence. 95% confidence is a standard and really only this should be accepted.