A/B Testing

Definition

Comparing two versions under controlled conditions to isolate a tactic’s true effect. A hotel can rarely split guests randomly like a website does, so RM adapts the idea: date pairing (matched test and control weekends), alternating weeks, segment splits, or a sister property as control.

What it tells you

“We introduced it and revenue went up” proves almost nothing — season, market and compset moved too. A controlled comparison separates the tactic’s effect from the environment. Two quick honesty checks replace formal statistics: do all pairs point the same way, and is the average effect large against the noise (the spread between pairs)? A signal-to-noise ratio above ~3 is a strong result; below ~2, treat it as noise.

How to track it

Write the hypothesis, the metric and the decision rule down before the test starts, run it for the pre-agreed window, and log the result either way. The classic failure mode is retrofitting: every test looks successful if you choose the metric afterwards.

Where it fits

The discipline that turns pricing opinions into evidence — and the natural input of the decision log. The free Academy covers it in depth: A/B testing and revenue experiments.