How intelligent is your A/B testing?

Jozo KovacGeneral

Multi-armed bandit ab tests

Everyone in marketing loves A/B testing.  It’s a simple, brilliant idea: Everything can be improved if more variants are tested. It is especially loved in ecommerce as marketers test designs, claims, call-to-actions over all available communication channels.

But then there is the down side. By definition, many A/B testing variants yield sub-optimal results. The longer bad variants are tested, the higher the losses will be.

Can we reduce the A/B pain?

Is it somehow possible to make A/B testing easier, more automatic, and minimize losses in the process? Yes, we can and this is where we can get some help from AI, probability theory, and the set of algorithms called Multi-Armed Bandits.  (

The idea comes from gambling on row of slot machines – Which lever should I pull to get the highest reward? Does it matter what order I play the machines? What’s the final payout going to be?  It’s the same as in A/B testing – Which testing variants should be used to get the highest rewards?

Do you trust the algorithms?

An algorithm is only good if it can be tested and verified. We’ve made a series of tests to see if the AI with the multi-armed bandit deserved our trust. Here’s the summary:

Can AI find it if we make variant A better than B? YES – and it was very, very fast. After a while, it gave a 99% weighting to A. 

1 (1)

Can AI react if we switch the coin and make B better? YES – the remaining 1% of the B variant helped AI to identify that B was getting the better numbers. It was subsequently tested more and ultimately was appointed as the winner.

2 (1)

Can we trick AI with two identical variants? NO — AI quickly found the balance and showed that both variants scored equally.

3 (1)

The results from this test convinced us that we could trust AI to optimize A/B testing, opening up a really wide range of options for Exponea.

Test drive the new A/B Bandit in the sky

Following our tests, we have implemented an advanced AI variation of Multi-Armed Bandits algorithm in the Exponea marketing cloud. I believe it will change the way A/B testing is run. As an Exponea user, you simply need to decide what goal to optimize: It can be a purchase, a pure click, or any other event such as registration, subscription to a specific service, and opening emails. There’s no longer a requirement for you to define the precise percentages for all variants. Just create your choice of variants and let the algorithms choose the best ones.

For more information on putting this new intelligent A/B testing function to work, just talk to your Value Delivery person or a sales rep.