How to Formulate a Solid and Reliable A/B Testing Hypothesis

Introduction

If you've ever sat with your team discussing why the sign-ups are low or why the conversion needle is almost never moving, then it's most likely that you are discussing a hypothesis problem. Hypothesis is the base of why you conduct an A/B test. You find an issue, and then you form a hypothesis regarding how to fix the issue. Without a proper hypothesis, you're just guessing away at your conversion problems, and guessing rarely leads to success.

What Is an A/B Testing Hypothesis?

An A/B testing hypothesis can simply be defined as making an 'educated guess' on a specific theory or change to see how it impacts user behavior or performance metrics. You can also think of it as a statement that predicts how an A/B test will perform, or what results the test would bring.

For instance, imagine you're testing the subject line of your festive sale marketing email. You 'assume' or 'make a guess' or 'predict' that having a subject line like — 'Hey, there, did you check your 20% discount coupon' — can increase your opening rates because it is more personalized and comes with an exciting offer. This is your 'hypothesis.' It predicts the outcome (higher opening rates) and also provides a rational explanation (more personalization) as to why it would work.

Your hypothesis can be anything — maybe changing the CTA button size can increase conversions, or making the headline shorter, changing the CTA wording, or adding a video. Hypothesis gives your experimentation processes a proper start, without which you may be conducting random tests with no means to measure success. The best part about a hypothesis is that it ensures you learn something valuable regardless of whether the hypothesis turned out to be a success or not: if it is correct, you have found a change that works; if it fails, you gain insights into what's not working.

Building a Strong A/B Testing Hypothesis

1. Rely on Data and Not Guesswork

Consider the difference between these two hypotheses: 'Increasing the size of the CTA button could increase conversions' versus 'Increasing the size of CTA by 20% could increase conversions by 5%.' The latter is clear, more reliable, and eliminates guesswork. Data is arguably the best way to ensure your hypothesis is solid, has a good chance to bring in positive changes, and eliminates guesswork. Dive deeper into Google Analytics, heatmaps, session recordings, interviews, forms, and more to understand user behavior and spot pain points.

For instance, if you discover that more than 50% of customers do not move from the second-last step to the checkout page, you have a basis for a hypothesis. On thorough analysis, if you realize that a trust factor is missing on the page, you can form a specific, data-backed hypothesis: 'Adding rating and social proof can reduce page bounce rate by up to 10% and increase conversions by 7%.' The numbers come from data and data alone — not random guesswork.

2. Be Clear and Specific

Instead of going in circles, you are better off being super clear and specific in your hypothesis. A vague hypothesis like 'Maybe changing the CTA button color and the image size a bit could increase business' fails on multiple counts: 'maybe' signals uncertainty, 'image size a bit' is unmeasurable, and 'increase business' sets no clear success metric. By contrast, 'Changing the CTA color from blue to red can help boost conversions by 6%' is clear — the element in question (CTA), the change (blue to red), and the expected result (6% conversion increase) are all explicit.

3. Avoid Multivariate Testing at the Beginning

It can be tempting to test many variables together to get faster results, but testing too many variables together makes it difficult to spot which element was actually creating friction and which change was bringing in results. For instance, if you simultaneously move the CTA button to the center and add a video, and conversions rise by 13%, you cannot determine which change confirmed your hypothesis. Making a change by keeping all other elements unaltered and removing any external disturbance is the best way to test your hypothesis for its worth. Multivariate testing is a valuable methodology once you have solid data analysis to rely on, but beginners should start with isolated changes.

4. Keep It Actionable, Simple, and Testable

Your hypothesis should be simple, to the point, and something that can actually be tested in real life. Predicting that changing the font of your website can impact conversion rates is very vague, and many CRO experts would agree that font is not that important a factor. In similar cases, you risk wasting time and resources on hypotheses that most likely would yield nothing. It is thus super important that your theories are actionable and have the potential to bring in a positive business impact.

Key Components of a Strong A/B Testing Hypothesis

Certain components ensure your hypotheses are strong, actionable, solution-oriented, and clean. Integrating all three — problem, solution, and outcome — solidifies your hypothesis generation and gives you a better chance at success.

1. Detecting a Problem

The first component required for any hypothesis formulation is detecting a problem. This could involve deeply analyzing common metrics like conversion rates, CTRs, bounce rates, average session duration, cart abandonment rate, and more. Examples: 'Our website sign-up rate is 3% lower than the typical industry standards' or 'Our cart abandonment rate is 20% higher than competitors.' Once you locate a problem through generic analysis or data deep dives, you have formed a base for your hypothesis.

2. Presenting a Solution

Once you gain insights into the problem, you can narrow your focus quickly to address it. Outline the specific changes or interventions you believe could help resolve the issue, ensuring the solution is actionable and measurable. For example, if data reveals the CTA button is extremely small or nearly invisible, you propose increasing its size. This proposed solution is what you will test against the existing variables.

3. The Outcome

Your outcome is where you outline what you expect from conducting the A/B test. For example, 'Increasing the sign-ups by 8%.' By specifying the outcome, you provide a metric for results to be tested against. If sign-ups increased only by 2%, you know something went wrong — whether it was the hypothesis, the element, or the chosen problem and solution. It could be that the CTA was never the problem; maybe it was the absence of social proof.

Where to Find Hypothesis Ideas

Data is your true best friend for generating A/B testing hypotheses — there is arguably no better place to detect anomalies and find customer pain points. The minute you convert a pain point into a hypothesis, you positively boost your chance of higher conversions. But if your data is not telling a story, start looking around: what problems do you typically face when you use an app or website, and is that problem present on your page too? Often, the problem is right in front, but our biases can prevent us from seeing the gaps and inconsistencies.

Competitor analysis can be another excellent source — conduct deep audits of what's working for your competitors and what's not. Academic papers, articles, and case studies are sometimes goldmines for hypothesis ideas. Don't just stick to one field; even if you come from the SaaS industry, understanding how the eCommerce industry formulates and tests its hypotheses can spark curiosity and help test unique ideas. Conversations with peers, customers, experts, and even those outside your field can also surface challenges and curiosities that lead to a groundbreaking thesis. AI and technology tools — testing different APIs, prompts, and more — can conduct deeper analysis and provide a continuous flow of fresh ideas.

Testing, Measuring, and Iterating

Formulating a hypothesis is the first step of the entire A/B testing process. Testing, measuring, and iterating are all integral to understanding what's working, what's not, and how to improve.

• Testing: This is putting your hypothesis in motion. You create two versions — variation A (the original) and variation B (the change) — and present both to a set of audiences.
• Measuring: Once the test is completed, you collect and analyze the data. How many people clicked on the CTA button? Did variation B perform better than A, and if so, by how many percentage points? Measuring results is the ultimate test of your hypothesis.
• Iterating: Based on the results, you understand what to do next. If variation B worked, you can implement it or push the experiment further. If not, you can tweak your hypothesis and test again — maybe it was not the size but the color of the button.

A/B Testing Hypothesis Example: HubSpot

HubSpot Academy's homepage was underperforming. On analyzing data, HubSpot found that only around 0.9% of the 55,000 visitors were actually spending time on the homepage video, and messaging was all over the place. HubSpot deployed 3 variants — A (controlled version), B, and C. Variant B included more colorful text and images and an animated headline. Variant C experimented with placements of the headline and images. Variant B outperformed Variant A by almost 6%, which translated to 375 more sign-ups for HubSpot. Variant C underperformed by 1%.