Hypothesis Testing & A/B Tests
Hypothesis testing lets you decide whether an observed difference is real or just random noise. Learn the null hypothesis, p-values, common tests, and how to run a sound A/B test.
The Logic of a Test
A hypothesis test compares two competing claims:
Null hypothesis (H0) — there is no effect or difference.
Alternative hypothesis (H1) — there is an effect.
The p-value is the probability of seeing data at least this extreme if H0 were true. A small p-value (commonly below a pre-set significance level alpha of 0.05) is evidence against H0.
Choosing a Test
The right test depends on your data:
- t-test — Compare means of two groups (numeric outcome).
- Paired t-test — Same subjects measured before and after.
- Chi-square test — Association between two categorical variables.
- ANOVA — Compare means across three or more groups.
- Mann-Whitney U — Non-parametric alternative when normality fails.
Example: Comparing Two Groups
Did a new checkout page change average order value?
import numpy as np
from scipy import stats
rng = np.random.default_rng(42)
control = rng.normal(loc=52, scale=12, size=400)
variant = rng.normal(loc=54, scale=12, size=400)
t_stat, p_value = stats.ttest_ind(variant, control, equal_var=False)
# Effect size (Cohen's d)
pooled = np.sqrt((variant.var(ddof=1) + control.var(ddof=1)) / 2)
d = (variant.mean() - control.mean()) / pooled
print(f"t = {t_stat:.2f}, p = {p_value:.4f}, d = {d:.2f}")
if p_value < 0.05:
print("Reject H0: difference is statistically significant")Running a Sound A/B Test
Good experiments are designed before data is collected:
- Pick one primary metric and the minimum effect worth detecting up front.
- Calculate the required sample size from power (usually 80%) and alpha.
- Randomly assign users and avoid stopping early when results "look good".
- Report effect size and a confidence interval, not just the p-value.
- Beware multiple comparisons — testing many metrics inflates false positives.
Key Takeaways
- A p-value is not the probability that H0 is true.
- Statistical significance is not the same as practical significance.
- Decide sample size and metrics before running an experiment.
- Always report effect size alongside the p-value.
Contact Us
Have a question or feedback? Fill out the form below or reach us directly at support@nvaitraining.com