Analysis Paralysis
Menu
← All categories

Category

Experiments & Evidence

A/B testing, statistical reasoning, and what the evidence actually allows us to conclude.

For anyone who wants to understand both the power and the limits of an experiment.

Reading guide

What questions are we asking?

A statistically tidy result can still be a poor basis for a decision. Experiments depend on the people, period, comparisons, and objectives you choose. This guide helps you ask how far a result travels before treating a winning variant as a lasting answer.

  • When does an A/B test result stop being relevant?
  • Does beating one variant establish an overall ranking?
  • What do repeated local wins leave unexplored?

A suggested reading path

  1. Statistics and analysts

    Build the habit of checking assumptions and how the sample was produced.

  2. The Temporal Limitations of AB Testing

    Ask whether a result from one period applies to the next.

  3. The Intransitivity of AB Tests

    Understand why separate comparisons need not produce a global winner.

  4. The Greediness of AB Tests

    Consider what the search for incremental wins prevents you from learning.

Try it at work

Choose a completed experiment. List its population, time window, primary metric, and comparison. Name one change in context that would make you test the conclusion again.

Series

The Limitations of A/B Tests

A short series on how experimental results fail when they ignore time, transitivity, and greed for local wins.

Explore the series →

All essays in this category