Thank you!!

hey Mani!

These are all great questions… unfortunately, there are no simple answers to them.
So, I’ll write a whole article about this topic soon! Stay tuned! 😉

Tomi

Each software provider seems to use their own approach: g-test, t-test, z-score, Chi square, one vs two sided tests, sequential testing vs fixed horizon, etc. Is there a preferred approach for marketing and product teams?

Relatedly, power calculations using online calculators may not match the statistical approach of the A/B vendor. Does that matter materially?

It’s easy enough to just defer to the software, but I’d appreciate your input.

hey Stephen,

great question and it’s a common one, too.
There are different opinions on this… I can only give you mine (it more or less applies for online business only):

If you have a small audience or a small number of conversions, then don’t A/B test at all.

Here’s an example:

Let’s say, you have an e-commerce shop with monthly ~2.000 visitors and ~5 sales.
If you were running an A/B test, you should literally wait months to get significant results.
(Except if you test something very impactful.)

In that case, I’d not spend a minute (or dollar) with A/B testing.
I’d first go for increasing the traffic first. (Put money and time into building marketing.)
At that level, it’s really easy to double/triple the audience and the sales with it, too.
With an A/B test your realistic goal would be to add only +20% to conversion… So it’s 200% vs 20% at this size.

Of course A/B testing is not only about increasing conversion, it’s also about understanding your audience.
But at this size, it’s not the best research method, either.
I’d recommend to use:
– more usability testing / user interviews (https://data36.com/usability-testing-data-analysts/)
– basic analytics methods (heatmapping, google analytics, etc.)

And most probably you’ll find super simple issues (button on wrong place, missing links, etc) that you can change without even A/B testing it — and you will still increase your conversion with a very high certainty.
Plus, since the sales is low, your risk will be low, too…

Well, this is only my opinion on the topic.
Hope that it makes sense and that it helps.
(If I misunderstood your question, let me know!)

Tomi

Thanks a lot!

Thank for the amazing article here.

It’s very tricky to explain Statistical Significance and I think this was the ideal approach. I have been looking for a way of ‘humanising’ the concept. I think the comparison between viewing the data from a business point of view and an ’emotional’ gambling perspective makes it easy to relate to.

This has given me the basis for reiterating the importance of testing and means for pushing back on hasty business decisions.

What is still challenging is balancing the need for Statistical Significance and working with low volume projects. Some sources cite a need to reduce reliance on Statistical Significance to push forward outcomes for low volume tests. I’ve also read about balancing data with principle based thinking which would make sense when you think about providing value to a project with limited data.

What would you say about Statistical Significance and low volume data? In my view, it makes it more difficult as running a test with Statistical Significance gives you a stable platform to work from. I also think it pushes the tester to learn more about the ins and outs of Statistical Significance as they will have to master how to get around the disadvantage of not having data (i.e. your first example vs. your second) in optimising a product or experience.

Cheers!

Stephen