What are A/B Test Results & How to Interpret Them?

Analyzing your A/B results is arguably the most crucial stage of an AB test. Despite this, it's one of the least discussed aspects of AB testing.

However, before we get into that, it's important to touch upon two critical metrics. These include:

• Uplift: This refers to the difference between the performance of the element getting tested and the performance of its baseline version (this is typically the control group). For instance, if one version has a revenue per user of $5, and the baseline version has a revenue per user of $4, the uplift in this case would be 25%.  
  
  

• Probability To Be Best: This refers to the likelihood of a version having the best long-term performance. In other words- the version that wears the crown in your A/B testing results report. It’s important to note that the Probability To Be The Best metric doesn’t begin calculating unless there have been 30 conversions or at least 1000 samples. 

One good way to understand the above two metrics is that Probability To Be Best metric answers- Which version is better, and Uplift metric tells you by how much.

Now, when interpreting AB results, you need to make your way through three levels. Here they are, in chronological order: 

1.Basic Analysis

Once you receive your AB testing results, the first thing you should do is check whether your A/B testing results have a winner and whether the results are statistically significant. 

Statistical significance in AB results refers to ascertaining the probability that the AB testing results are not due to chance and depict the accurate difference between the two tested versions. 

Typically speaking, your A/B test results’ winner will only be determined if the following conditions have been fulfilled:

• One of the two versions has a Probability To Be Best score of 95% or higher (you can change this standard using the winner significance level or similar setting on your chosen A/B test tool).

• The test has run for the specified minimum duration (usually two weeks—but it can be tweaked to ensure the A/B test results aren’t compromised due to seasonality.

Now, compare the baseline version’s performance to the challenger version’s. Your winner would have performed better on the Key Performance Indicators (KPIs) that you’re aiming for.

2.Secondary Metrics Analysis

The basic analysis method takes primary metrics into account, such as your conversion rate or revenue per user. The secondary metrics analysis method, as the name suggests, factors in secondary metrics (engagement metrics, return visitor rate, cart abandonment rate, etc.) that might not be part of the AB testing goal but are nonetheless important to consider.

Taking the time to perform this analysis offers the following benefits:

• Help you avoid mistakes

Performing secondary metrics analysis helps you avoid getting blindly carried away in the celebrations of driving your primary metric. For instance, your winning version might have performed well as per your primary metric- Click-Through Rate (CTR), but instead cost you your revenue or Average Order Value (AOV). 

With a secondary metrics analysis, you get a more balanced picture of your winning version’s performance.

• Uncover interesting insights

Digging deeper into your AB test results with secondary metrics can help you get other interesting insights that might not be apparent on the face of the results. For instance, let’s say your AB testing results find that for the winning version, the purchase per user fell, but the AOV rose. 

This could mean that your winning variation prompted users to purchase less in quantity but purchase more expensive products, driving revenue. Something you wouldn’t happen upon if you didn’t undertake secondary metric analysis.

Pro tip: Analyze your Uplift and Probability To Be Best scores for each secondary metric to understand how each version performed. This will tell you whether you can serve all your traffic with the winner version or instead tweak your allocation depending on what you’ve uncovered.

3.Audience Breakdown Analysis

The final analysis that helps you truly wring every last drop of value out of your AB testing results involves segmenting your audience by behavior, demographics, or any other relevant factors. Doing this allows you to answer questions such as:

• How did the traffic source affect the test results?

• Which version won for desktop, and which one won for mobile?

• What version works best for new users?

Now, while it can be tempting to segment your audience mercilessly to drive personalization, keep the following in mind during segmentation:

• Keep segments large enough to ensure statistically significant results

• Keep segments relevant to your business goals

• Keep segments actionable in terms of personalization efforts or any other future strategy

Again, for every audience, analyze the Uplift and Probability To Be Best metric scores to ascertain how every version performed. This will help you determine whether you should serve the winning version to all your traffic or tweak it based on your learnings.

Make your way through the A/B test results according to the above and you can rest assured that you’re getting the most out of your AB testing results. 

But wait, there’s more.

At the beginning of this section, we’ve presumed that your AB testing results have found a winner. But what if your test doesn’t come with a winner? What happens then? 

Is the test redundant? Does all hell break loose? 

No, and no. You see, as personalization takes center stage with customer engagement and conversions, tests and experimentations that don’t consider the distinct individual audience conditions often conclude with no winner, with statistical significance becoming difficult to achieve.

In other words, the usual one-to-many testing approach will not work for all visitors anymore. There will always be a portion of your audience that your winning version will not address. 

Once you understand this significant drawback of AB testing, you come to the conclusion- that AB tests with no winners might actually have winners, too. 

Let’s take an example to understand this better.

In the following AB test that ran for 30 days, the AB testing results reported the control version outperforming the challenger version.

With this, you’ll naturally believe the challenger version is just sub-par. But break the test down by devices, and you uncover a completely different picture. 

Breaking the test down by user devices, the controller version wins on desktop, but the challenger version outperforms it on tablet and mobile.

This example is meant to underscore the importance of dissecting your A/B test results. Only once you’ve analyzed your AB results can you confidently move to implement the conclusion of the test.

But what if we told you, you could make all this easier and quicker? All you need is a tool like Fibr AI. With a powerful reporting and analytics feature that gives you an in-depth analysis of your campaign performance, Fibr AI helps you get instant insight into which variation is your top performer. As a result, fueling quicker optimization, informed decision making and overall campaign efficiency.