Data scientist interview: an A/B test shows a 2% lift but it is not significant — what do you do?

Model answer

Resist the urge to ship on a flattering number. Start by stating the principle: a 2% lift that is not statistically significant means you cannot rule out that the difference is noise, so you should not yet claim the change works. Then show what you would actually do. Check the test's health first — was the sample size and duration enough to detect a 2% effect? A common cause of "not significant" is being underpowered, so you might need to run longer or with more traffic. Verify the setup — proper randomisation, no sample-ratio mismatch, and consistent metrics. Consider whether 2% even matters practically; if the effect is real but small, weigh it against the cost of the change. Options include continuing the test to reach adequate power, iterating on a stronger version of the change, or accepting no difference and moving on. Be honest with stakeholders about uncertainty rather than over-selling. Avoid two mistakes: shipping just because the number is positive, and stopping the moment it looks good. The interviewer is testing statistical discipline and integrity, so emphasise deciding with proper evidence.

This is a guide to structure and approach, not a script to memorise — say it in your own words.

Practice this yourself

Type your own answer below and get it scored instantly by the same AI that powers Vachix mock interviews.