How does A/B testing work?

A/B testing (also called split testing) is a method of comparing two or more versions of a page, feature, or UI element to determine which performs better. You randomly split users into groups, show each group a different variant, measure a specific metric (conversion rate, click-through rate, engagement), and use statistical analysis to determine which variant wins. It removes guesswork from product decisions by letting real user behavior decide.

The core process follows a scientific method: form a hypothesis, design an experiment, collect data, and draw conclusions. Here's the typical flow from start to finish.

On the frontend, A/B testing typically involves assigning users to a variant (usually via a cookie or user ID hash), conditionally rendering the appropriate UI, and tracking events. Here's a simplified implementation showing the core pattern.

type Variant = 'control' | 'treatment';

function getVariant(experimentId: string, userId: string): Variant {
  // Deterministic assignment — same user always gets same variant
  // Hash the combination of experiment + user to get a stable 0-1 value
  const hash = simpleHash(`${experimentId}:${userId}`);
  const normalized = hash / MAX_HASH_VALUE; // 0 to 1

  // 50/50 split
  return normalized < 0.5 ? 'control' : 'treatment';
}

function useExperiment(experimentId: string): Variant {
  const userId = useUserId(); // From auth context or anonymous ID
  const variant = useMemo(
    () => getVariant(experimentId, userId),
    [experimentId, userId]
  );

  // Track exposure (user saw this variant)
  useEffect(() => {
    trackEvent('experiment_exposure', {
      experimentId,
      variant,
      userId,
    });
  }, [experimentId, variant, userId]);

  return variant;
}

The key requirement is deterministic assignment — the same user must always see the same variant across page loads and sessions. This is typically achieved by hashing the user ID with the experiment ID, giving a stable result without needing to store the assignment.

function SignUpButton() {
  const variant = useExperiment('signup-button-color-2024');

  return (
    <Button
      variant={variant === 'treatment' ? 'default' : 'secondary'}
      onClick={() => {
        trackEvent('signup_clicked', { variant });
        handleSignUp();
      }}
    >
      Sign Up Free
    </Button>
  );
}

You can't just look at raw numbers and declare a winner. If variant B has a 5.2% conversion rate vs A's 5.0%, that might be random noise. Statistical significance testing tells you whether the observed difference is likely real or just chance. The standard threshold is a p-value below 0.05 (95% confidence).

This question tests whether you understand data-driven product development. Interviewers want to see that you know how to design an experiment, understand the importance of statistical rigor, can implement feature flags and variant assignment on the frontend, and appreciate the pitfalls (peeking, insufficient sample size, novelty effects). It shows you think beyond just building features — you think about measuring their impact.