A/B testing and revenue experiments

Early June, Monday afternoon. Daniel arrives at the meeting with a finished plan: “Since the May 26 concert weekend we know what MLOS-2 could be worth (lesson 42) — in the model, the three days would have beaten the no-restriction path by 30%. I propose we extend it: the two-night minimum goes out on every strong Saturday, and we raise the weekend BAR ladder alongside it. Over the summer and autumn together, that is easily a five-figure plus.”

Adam leans back. “I like it. One question: how will we know it worked? Last autumn we switched to the new weekend pricing too; fourth-quarter RevPAR grew six percent — and to this day we argue about whether that helped or the market simply came back. The compset grew then as well. I’d rather not have two opinions again next year instead of one shared number.”

Daniel smiles, because Adam is right — and because for this question a methodological answer exists. This lesson is about how to introduce a new revenue tactic so that it ends not in a debate but in a measurement: control group, pairing, effect size and a simple significance intuition. In lesson 46, with promo measurement, we already saw the core (incremental against a control) — now we build a general experimental toolkit out of it.

Why “we rolled it out and it grew” lies

In the world of e-commerce, A/B testing is trivial: a randomly selected half of the visitors sees version A, the other half version B, at the same moment, in the same market — the only thing that differs between the two groups is the treatment (the change being tested). The effect can therefore be read cleanly out of the difference. In a hotel this luxury does not exist, for four reasons:

• You cannot sell the same night twice. The May 26 Saturday either had the MLOS or it didn’t — there is no parallel universe where the same day runs the other variant.
• The sample is small. 80 rooms, 365 days. An e-commerce site measures on a hundred thousand visitors in a week; a hotel has maybe 20 strong Saturdays in a year. Every observation is expensive.
• The days are neither independent nor alike. A concert Saturday and a November Saturday are not comparable; Friday affects Saturday (the stay pattern spans across); the demand of two adjacent weeks is moved by the same market wave.
• The market itself moves. Season, compset pricing (lesson 44), events, the economy — everything changes, continuously, independently of your intervention.

The fourth point is the deadliest, because it kills the most obvious “measurement”, the before/after comparison (“we introduced it in September; October came in better”). Before/after doesn’t measure your tactic’s effect; it measures your tactic’s effect plus everything that happened meanwhile — the season turning, a market upswing, a competitor closing for renovation. This is exactly Adam’s last-year debate: of the +6% RevPAR, how much was the new pricing and how much the returning market? Before/after cannot answer that even in principle.

The control group: measuring “what would have happened”

The logic of the solution is the same in every experiment. The “does it work?” question really reads: how much better did the result get compared with what would have happened without the intervention? This “what would have happened” — in the trade, the counterfactual — cannot be observed directly. (Lesson 42’s scenario “B” — what the concert Saturday would have done had the MLOS-2 gone up in time — was exactly such a thing: a counterfactual computed from a model.) The control group replaces it in observable form: a comparison base that shares everything with the test group — season, market, demand environment — except the treatment. If there is a difference between the two, it can (in a good experimental design) only come from the intervention.

An e-commerce site achieves this with a random visitor split. A hotel has to form its control differently — and it has five practical methods.

Five ways to form a control in a hotel

A few notes on the methods:

Date pairing is the hotel RM’s workhorse. The pairing criteria: same day of the week, same season phase, same event status (concert with concert, event-free with event-free), same school-holiday status. Within the pair, decide by lot which date gets the treatment — don’t give it to the “suspiciously stronger” day, because that builds in exactly the bias the control is meant to filter out.

Alternating weeks draw their power from the time trend averaging out: if the market happens to strengthen during the test, the strengthening lifts the on and off weeks roughly equally and drops out of the difference. Watch out for spillover: if a guest turned away in a treatment week rebooks into the following — control — week, the control is no longer untouched.

The segment split is the one place where a hotel gets e-commerce-grade randomisation: a random half of the newsletter list (lesson 49) gets the new member rate or offer copy, the other half the old one — at the same moment, in the same market. If you have a closed circle, test every closed-circle tactic this way.

The synthetic control is the generalisation of lesson 46’s cannibalization estimate: there, the historical pace said how many bookings would have come without the promo — here, the historical curve and the market trend together give the reference path for any intervention. Fine as a last resort; when a real control can be formed, that is always stronger.

The MLOS test in numbers

Back to Daniel’s proposal. The answer to Adam’s question is an experiment plan — and the first discipline decision hurts immediately: Daniel originally wanted two changes at once (the MLOS-2 extension + a higher weekend BAR ladder). If they go out together and revenue grows, we never learn which one worked — and if it falls, neither which one hurt. One variable at a time: MLOS-2 goes into the test first; the BAR ladder is the next experiment.

The plan (fixed in writing before launch):

• Hypothesis: a stay-through MLOS-2 on strong Saturdays, managed with a timed release (lesson 42’s sequence), increases weekend-block revenue.
• Sample: the next 8 weeks’ 8 strong Saturdays (each with 85%+ forecast Saturday demand). Paired by event status: 1 concert pair, 1 trade-fair/conference pair, 2 event-free strong leisure pairs. Within each pair, a coin flip decides which date is the treatment.
• Metric: the 3-day weekend block’s (Friday+Saturday+Sunday) room revenue — lesson 42’s lesson: measure the weekend, not the Saturday, because the MLOS’s benefit is realised on the shoulder days.
• Decision threshold (in advance!): success if the average pair difference reaches +800 EUR and at least 3 of the 4 pairs are positive. Below that: no extension.
• Everything else equal: the same pricing logic runs on all 8 weekends, no promo on any of them.

Eight weeks later, the result:

The effect size: on average +1,630 EUR per weekend, which against the control weekends’ 16,275 EUR average is +10%. Counting ~20 strong Saturdays a year, the tactic’s potential is in the order of 32,600 EUR a year — that is the number that can go onto Adam’s desk.

Signal or noise?

But before we celebrate: 4 pairs is very few. Could the +1,630 be plain luck — could noise (the natural, random fluctuation between pairs) produce a difference this size? Instead of a formal statistical test, two simple checks give the intuition:

1. The direction check. All 4 pairs point the same way. If the MLOS had no effect at all, each pair would come out positive or negative with 50–50 odds — the chance that all four are positive by accident: 0.5⁴ = 1/16 ≈ 6%. Not impossible, but already improbable.

2. The signal-to-noise ratio. The signal is the average effect; the noise is the standard deviation of the pair differences (how much they fluctuate around the +1,630 average). The four deviations from the average: +490, −40, −340, −110 → the standard deviation is ~351 EUR. The ratio:

signal/noise = average difference / standard deviation of the differences = 1,630 / 351 ≈ 4.6

Rule of thumb: below 2, the “effect” can easily be noise — don’t decide, keep running; between 2–3, promising, but worth collecting more pairs; above 3, the signal is strong. The 4.6 says: the measured effect is more than four times the usual pair-to-pair fluctuation — noise practically does not produce a difference this size. The pre-agreed threshold (≥ +800 EUR, ≥ 3 positive pairs) is met with room to spare: the tactic can be extended.

And the honest footnote: had the signal-to-noise come out at 1.5, the right decision would not have been reinterpretation (“still, something shows on the concert pairs…”) but an extension of the run — 4 more pairs, then look again. A small sample is no shame in a hotel; a confident conclusion drawn from a small sample is.

The rules of experimental discipline

The test above is good not because of the arithmetic but because of the discipline. Five rules — breaking any one of them renders the whole measurement worthless:

1. Fix everything in advance: hypothesis, metric, time frame, decision threshold. If what counts as success isn’t stated up front, then afterwards every experiment is successful — you will always find a slice (a segment, a date window, a metric) where the number looks good. Lesson 46’s promo lesson, generalised: the definition of success is born before launch, not after.

2. One variable at a time. The effects of two changes pushed out together are inseparable. If you want to test two things, that is two experiments — in sequence.

3. A minimum run time — don’t stop after 3 days. Neither in triumph (“it already shows, roll it out!”) nor in panic (“the first weekend is weaker, switch it off!”). The first pair is always noise. The run time is filled by the sample size fixed at launch (here: 4 pairs) — stop early only if the tactic is doing active damage.

4. A negative result is a result too. “MLOS-3 brought no surplus over MLOS-2” — that saves someone reintroducing it next year and the debate starting over. Document it: what we tested, when, what came out, what was decided. In lesson 64 we’ll see how this becomes a systematic decision log — the hotel’s institutional memory.

5. Know the business limits of experimentation. Don’t experiment with the peak revenue weeks (New Year’s Eve, the biggest fair week) — a failed treatment there costs too much, and they are too atypical for the lesson to generalise anyway. Don’t run risky price tests on the regular-guest circle — lost trust doesn’t come back when the test is switched off. And every treatment should be reversible: a restriction can be lifted; a publicly announced price promise is much harder.

When is an experiment needed — and when not?

An experiment isn’t free: planning, waiting, and the possible revenue sacrifice of the tactic “not applied” on the control dates. So it isn’t for everything:

• No experiment needed for trivial and easily reversible decisions. Fixing a mistyped rate, removing an obviously forgotten restriction, a 2 EUR price fine-tune along the system’s recommendation (per lesson 36’s elasticity logic) — just do it, and watch the pickup. Here the cost of the experiment exceeds the stakes.
• An experiment is due for lasting, hard-to-reverse, contested or high-stakes changes: a new price position against the compset, a new channel strategy, a restriction system (not one day, but a rule for every strong Saturday), the pricing of a member programme. What they share: if you are wrong, it costs a lot, and the “does it work?” question would be debated anyway — the experiment is cheaper than running a bad strategy for a year or arguing about a good one for a year.

Between the two lives the grey zone — and there a useful in-between form: the evaluated rollout. You don’t form a physical control, but you fix the metric and a synthetic control (historical curve + market trend) in advance, and set a review date. The same idea powers the retrospective evaluation of pricing-recommendation overrides (lesson 56): every override is a mini-experiment — the system’s recommendation is the (synthetic) control, the manual decision is the treatment, and afterwards you can check which one would have won.

Back on Adam’s desk

Nine weeks after the June meeting, Daniel puts a single page in front of Adam — by lesson 60’s recipe, leading with the number: “On the tested strong Saturdays, MLOS-2 brought +1,630 EUR per weekend, +10% against the control. All four pairs positive; the signal is more than four times the noise. I propose extending it to every Saturday above 85% — and we measure the BAR-ladder raise in a separate test in the autumn.”

Adam runs through the pair table, then asks last year’s question: “And how do we know it wasn’t just the market being strong in those weeks?” — and this time there is an answer: “Because the control Saturdays lived in the same market. If the market had lifted, it would have lifted the control too — the market drops out of the difference. That is exactly why we didn’t measure against last year, but against each other.”

Last year’s debate — did it help, or did the market come back? — does not repeat this time. Not because Daniel argues more convincingly, but because he built the answer machine for the question before the launch. That is the real yield of A/B thinking in revenue management: not more data — decidable questions.

Key takeaways

• Before/after comparison in a hotel almost always lies: it measures your tactic’s effect blended with the season, the market and the compset moving. Only a control group can answer “does it work?” — a comparison base that shares everything with the test except the treatment.
• A hotel is not an e-commerce site — you cannot sell the same night twice, and the sample is small — but it has five adapted control methods: date pairing, alternating weeks, segment/channel split (the only true randomisation — in closed circles), multi-property test and synthetic control.
• The effect is carried by the pair differences: in the MLOS test 4 pairs, average +1,630 EUR per weekend (+10%), all four in the same direction (chance of that with no effect: 0.5⁴ ≈ 6%). The significance intuition: signal-to-noise ratio = average effect / standard deviation of the differences (1,630/351 ≈ 4.6) — below 2 it may be noise, above 3 the signal is strong.
• Without discipline there is no experiment: pre-agreed hypothesis + metric + time frame + decision threshold; one variable at a time; the sample run to completion; documented negative results; and no experiments on the peak weeks or on the regulars’ trust.
• Not every decision needs an experiment: trivial, reversible steps you just take. Experiments are owed to the lasting, hard-to-reverse, contested or high-stakes changes — there they are cheaper than running a bad strategy for a year or arguing about a good one for a year.