A/B Testing Fundamentals: How to Set Up Statistically Sound Landing Page Tests

A/B testing sounds simple: show two versions of a landing page, see which one gets more sign-ups, ship the winner. In practice, most teams get the mechanics wrong in ways that quietly invalidate their results. Understanding the statistical fundamentals before you launch your first test is what separates a testing program that produces reliable, compounding gains from one that ships false winners and loses stakeholder trust within a few months.

The good news is that the core fundamentals are learnable in an afternoon and, once internalized, apply to every test you'll ever run. You don't need a statistics degree — you need a handful of habits, applied consistently, that keep your process honest.

Define One Clear Primary Metric Before You Start

Every test needs a single primary metric decided in advance — typically sign-up form completions or a similarly unambiguous conversion event. Secondary metrics like time on page or scroll depth are useful context, but they should never be used to declare a winner if the primary metric is flat. Deciding this upfront prevents the common trap of hunting through dashboards after the fact for any metric that happened to improve, a practice statisticians call "p-hacking" that reliably produces misleading conclusions.

Calculate Sample Size Before Launching

Before a test goes live, use a sample size calculator with your current baseline conversion rate, the minimum lift you care about detecting, and your desired confidence level (typically 95%). This tells you roughly how many visitors each variant needs before the result is trustworthy. Skipping this step is the single biggest cause of false positives in landing page testing — small, low-traffic pages often need weeks or months to reach meaningful sample sizes, and that's simply the reality of statistical testing, not a flaw in your process. If your calculated sample size is unrealistic given your traffic, that's valuable information too: it tells you to either broaden the test's scope, extend the timeline, or apply best practices directly instead of formally testing.

Run Tests for Full Business Cycles

Visitor behavior varies by day of week, and sometimes by time of month for B2B audiences with different budget cycles. Running a test for only three or four days risks capturing an unrepresentative slice of traffic. As a general rule, run tests for at least one to two full weeks, and always in multiples of seven days, so weekday and weekend traffic patterns are represented equally in both variants. For businesses with pronounced seasonality, consider whether the test window includes an atypical period, such as a holiday, that could distort the result.

Avoid Peeking and Early Stopping

Checking results daily and stopping the moment a variant pulls ahead dramatically inflates your false positive rate — a well-documented statistical phenomenon sometimes called the "peeking problem." If you want to monitor results early, use a testing platform with built-in sequential testing correction, or simply commit to not making a ship decision until your pre-calculated sample size and minimum duration are both reached. It helps to write your stopping rule down before the test starts, so there's no ambiguity or temptation to rationalize an early call once the data starts trickling in.

A Simple Pre-Launch Checklist

• Primary metric defined and agreed upon by all stakeholders.
• Sample size and minimum duration calculated and documented.
• Variants implemented correctly and QA'd on both desktop and mobile.
• Tracking verified in a staging environment before traffic is split.
• A clear owner assigned to monitor the test and communicate results.

Choosing the Right Testing Tool

Most teams don't need to build custom testing infrastructure. Established A/B testing platforms handle traffic splitting, statistical calculations, and reporting out of the box, and many integrate directly with popular website builders and analytics tools. What matters more than the specific tool is consistent, correct usage — a simple platform used rigorously will outperform a sophisticated one used carelessly. For a deeper look at this, see our guide, Paid Traffic Channels.

Interpreting and Acting on Results

Once a test reaches its planned sample size and duration, evaluate the result against your primary metric and pre-agreed confidence threshold. A statistically significant winner should be implemented for all traffic; an inconclusive result is still useful information — it tells you the tested change likely isn't a major lever, freeing you to prioritize a different, potentially higher-impact hypothesis instead of re-testing minor variations of the same idea. Document every result, win or not, so your team's collective understanding of what moves your specific audience grows with every test. For a deeper look at this, see our guide, Programmatic SEO for Scaling Landing Pages.