Top 30 A/B Testing Interview Questions (2026)

Sample Answer

Sample size depends on four inputs: significance level α (typically 0.05), power 1−β (typically 0.80), minimum detectable effect (MDE), and metric variance. For a two-sample t-test: n ≈ 2·(z_α/2 + z_β)²·σ²/δ², where δ is the MDE. Estimate σ from historical data and let the product team define the MDE based on business impact. For proportions like conversion rate, use p(1−p) for the variance.

Sample Answer

An A/B test compares one control against one treatment. An A/B/n test compares one control against multiple treatments simultaneously. Use A/B for a single clear hypothesis. Use A/B/n when exploring several variants (e.g., 3 button colors), but apply a multiple comparisons correction (Bonferroni or FDR) to control the false positive rate. A/B/n requires more sample to maintain power across all comparisons.

Sample Answer

Guardrail metrics are metrics you don't expect to improve but must not degrade — like revenue, page load time, crash rate, or support tickets. They prevent you from shipping a change that improves your primary metric at the cost of something critical. Always define guardrails before the experiment starts, not after seeing results.

Sample Answer

Type I (false positive): you ship a feature that has no real effect — wasted engineering effort and potential hidden harm. Type II (false negative): you kill a feature that actually works — missed opportunity. Which is worse depends on context. At a payments company, Type I is worse (shipping a buggy checkout). At a social app testing engagement features, Type II may cost more (missed growth). This is why we tune α and β per experiment.

Sample Answer

Statistical significance means the observed effect is unlikely due to random chance (p < α). Practical significance means the effect is large enough to matter for the business. With a huge sample size, you can detect a 0.01% lift in conversion that's statistically significant but practically meaningless. Always look at the confidence interval and ask: is the lower bound of the effect still worth the engineering cost to maintain this feature?

Sample Answer

A p-value barely below 0.05 should trigger scrutiny, not celebration. Check if the test was properly powered. Look at the confidence interval — if it includes trivially small effects, significance doesn't matter. Check for peeking — if someone checked results multiple times, the effective α is inflated. Look at the metric distribution for outliers. Consider practical significance: even if statistically real, is the effect size worth the engineering cost? I'd recommend extending the test another week.

Sample Answer

A mean metric is per-user (e.g., average sessions per user). A ratio metric is an aggregate ratio (e.g., total purchases / total visits = conversion rate). The difference matters because the variance formula differs. For ratio metrics, you need the delta method to correctly estimate variance since numerator and denominator are correlated. Using a naive variance estimate leads to anti-conservative confidence intervals and more false positives.

Sample Answer

Several possibilities: (1) The clicks are low-intent — users click more but don't convert downstream. (2) Cannibalization — the new feature stole clicks from a higher-converting surface. (3) Novelty effect — users clicked out of curiosity but won't sustain the behavior. (4) The revenue metric has higher variance and the test wasn't powered to detect the revenue effect. Investigate by looking at the full funnel: click → add-to-cart → purchase. If clicks increased but downstream steps didn't, the extra clicks aren't valuable.

Sample Answer

A novelty effect is when users engage more with a new feature simply because it's new, not because it's better. The effect fades over time as users get used to it. Detect it by plotting the treatment effect over time — if the lift is large in week 1 and shrinks by week 3, it's likely novelty. You can also segment by new vs. returning users, or run the test longer (3-4 weeks) to see if the effect stabilizes.

Sample Answer

SRM is when the observed traffic split between control and treatment doesn't match the intended split (e.g., you expected 50/50 but got 52/48). Run a chi-squared test to check. Common causes: (1) the randomization code has a bug, (2) the treatment causes more crashes or redirects so users leave before being logged, (3) bots are disproportionately assigned to one arm, (4) the experiment interacts with another running experiment. SRM invalidates your results — you must diagnose and fix it before interpreting.

Sample Answer

It depends on whether the imbalance is random or systematic. Check if pre-experiment covariates (age, country, platform) are balanced between groups. If covariates are balanced and the imbalance is purely random, the test is valid — unequal splits reduce power but don't introduce bias. Adjust variance estimates for unequal sample sizes. If covariates are imbalanced, the randomization is broken and you must discard and re-run. A 60/40 split is suspicious enough to investigate the randomization code thoroughly.

Sample Answer

This is a covariate imbalance — a sign of potential randomization failure. First, run a sample ratio mismatch (SRM) test. If traffic counts are balanced but covariates aren't, check whether the treatment causes differential attrition (e.g., Android users dropping off before logging). If the imbalance is mild and random, you can adjust using stratified analysis or CUPED with platform as a covariate. If it's severe, investigate the randomization code — the experiment may be invalid.

Sample Answer

Frequentist testing gives you P(data | null hypothesis) — you set α upfront, collect data, and reject or not. You can't peek without inflating false positives. Bayesian testing gives you P(hypothesis | data) — you start with a prior, update with data, and get a posterior distribution over the effect size. Bayesian lets you peek freely and make decisions based on expected loss. Trade-off: frequentist has well-understood error guarantees; Bayesian requires choosing a prior and a decision rule. Netflix and some Spotify teams use Bayesian; Meta and Microsoft use frequentist.

Sample Answer

The delta method approximates the variance of a function of random variables — you need it when your metric is a ratio (e.g., revenue per session = total revenue / total sessions). Naive variance estimation treats each user independently, but ratio metrics have correlated numerator and denominator. The delta method uses a first-order Taylor expansion: Var(X/Y) ≈ (1/μ_Y²)·[Var(X) - 2·(μ_X/μ_Y)·Cov(X,Y) + (μ_X/μ_Y)²·Var(Y)]. Without it, confidence intervals are too narrow and you get inflated false positive rates. Companies like Google and Meta apply this automatically in their experimentation platforms.