Why AB testing can be a waste of money

Video transcript

Hi, this is Sander from Unless and today I will tell you why A/B testing doesn't work for most of us.

Over the past few years, A/B testing has been very popular, but recently it has lost its mojo. I will tell you why.

A/B testing is a form of split testing. It's usually about comparing one page to another, to see which one performs best - to see which one converts more visitors into customers woh buy something or become a member or so.

Usually, you can use the winner in your normal web flow to get more customers in. This doesn't work for everybody. I'll tell you why.

If you do any test, it only makes sense if you can be sure that it is correct.It means you have to reach "statistical significance". That is a fancy way of saying that you need to be able to rule out coincidence. Ruling out coincidence is actually very simple, but only for a really big improvement. For really small improvements it's VERY hard to prove if it's coincidental or not, and the problem is that conversion optimization is almost ALWAYS about the small improvements.

The math is super complex, but let me give you an example. If you want to increase your baseline conversion from 4% to 6% - which is a relative uptake of 50%, which is actually quite great - then, you need a staggering 1,600+ people to visit each variation. That still sounds reasonable, right? But let's say you want to increase your conversion rates from 4% to 4.4% - which is a relative uptake of 10% - which is way more realistic - then you need over 38.000 visitors on each variation!

And, you're not there yet, because let's say you performed your experiment during the week, then in the weekend it might well be that the rules of engagement for customers are entirely different, so you need to continue your experiment over the weekend to know for sure. And it's not just weekends: it could be seasonality, it could be the local news, it could be the Bitcoin crashing all of a sudden! It could even be the weather.

So typically A/B test has to be for much longer than you need, to just reach statistical significance.

So, what is the alternative for smaller companies?

If I get 100 people to my site and have a 5% conversion rate, that means I’d get 5 people to convert. I could try to get that conversion rate to 10% and get 10 people to convert. That would be a spectacular result.

OR I could just figure out how to get 1,000 new, relevant visitors. 5% of 1000 is significantly more conversions than 10% of 100. Less spectacular, but way more practical.

And you can spend your time only once, right? So, what should you spend your time on?

Personalization is the much better option. If you make your web pages specific for a certain audience, you will not only increase conversion, but if you segment your ads, too - you will get a better cost per click. Google rewards you for symmetry with an increased quality score with a better cps - and that translates to more visitors for the same money.

Spend your time wisely!