One subject that hoteliers ask us about time and again is how to test their website messaging and find out what performs best.
NB: This is an article from TripTease
For a long time A/B testing was considered the pinnacle of website testing, with countless messaging providers claiming to be able to offer an A/B service for hotels.
At Triptease we provide A/B tests for very few clients because truly reliable A/B tests are notoriously difficult to do. For most hotels their conversion volumes are too low to easily determine an uplift in conversion rate. It could take many months for a test to reach statistical validity – thousands of customers would need to convert on the website. Poorly set up A/B tests based on inappropriate hypotheses with inadequate amounts of traffic are all too common – and they waste hoteliers’ valuable time. You can read more on this subject in our 2018 A/B testing report for hotels.
Testing a hypothesis with message variants
So in place of unreliable A/B tests run by third parties, how can hoteliers test a hypothesis to improve the conversion rate of their website messages? One option is variant testing – a new feature we recently launched in the Triptease Platform.
Whereas A/B testing allows hoteliers to run a ‘control’ (a version of their website with no message on it) vs a ‘variant’ (a version of their website with a message on it), variant testing allows you to run two versions of the same message to see which one performs better.
While you’ll face similar issues around conversion volume and statistical validity as you would with an A/B test, variant testing means you can easily experiment with messages and closely monitor their performance over time. If you’re able to run a test for long enough to ensure you feel confident in the results then this is a great way to get a sense of what types of designs and messages your customers respond to best. It’s important to stay aware that the validity of the results will depend on a whole range of external factors, but variant testing will allow you to test ideas and gather conversion data to help back them up.
How to run a successful variant test
All good tests start with a hypothesis. It’s often helpful to try and format your hypothesis as a statement that you believe to be true and want to test. You would be testing that hypothesis against there being no difference between two variants, i.e. no effect.
Here’s a hypothesis template you could use as a starting point:
- I believe that (…) because (…)
- If I (…) then I will see (…)
In the first sentence you’re isolating the causal mechanism; in the second sentence you’re describing what would be monitored.
For example:
- I believe that a 10% discount is more effective than a 5% discount, because it’s a bigger saving for potential guests.
- If I run an experiment that does variant testing of a 10% discount vs a 5% discount, then I will see an uplift in clicks (or conversions).
It’s important to think about how any change you are proposing will lead to an impact. If the causal mechanism – in this example the discount rate – is not clear when thinking about variations, then it probably won’t have an impact.
It’s also important to allow your test to run over several weeks – ideally at least for the length of a booking cycle. The longer it runs for and the more data you gather, the more reliably you can make a call on whether or not your hypothesis has strong evidence. From here you can pick your ‘winning’ message variant in the Message Builder, and that message will continue to run on your website until you choose to switch it off.
