
What is Threatening Your Conversion Optimisation Test Results?

Your conversion optimisation tests have run, you’ve gathered your results, and now you’re analysing your data. It all looks good and you’re feeling pleased with how everything has gone. But, what tells you that everything is fine and that you’ll be able to trust your results?
You need to be aware of what threatens the validity of your tests. These problems are known as effects and they include:
Knowing what they are and by taking command, you’ll lessen their potential to do any damage.
When you insert incorrect tracking on your website you’ll suffer from an instrumentation effect. It’s a common error that harms results. But, you can avoid it by ensuring you take your time in setting up the test and by scrutinising the implementation with great care.
Importantly, ensure that you keep a very close eye on every goal and metric you want to track and confirm that they are being recorded by your analytics package.
If you find a problem (e.g., add to cart click data is missing), stop the test and fix it. Then reset the data and start again.
History effects occur when external factors interfere with your tests and challenge their validity. Think marketing campaigns, holiday seasons—Christmas, Easter, etc., the various seasons themselves, good/bad PR involving your company or industry, product recalls, and so on.
For example, you may remember in 2013 the media reported that breast implants made from unauthorised silicone fillers were being used by cosmetic surgeons. That made PIP a well-known company for all the wrong reasons.
And, any cosmetic surgery business in the UK doing conversion optimisation at that time would have seen the history effect in their results.
A positive example would be if a famous person (maybe the Duchess of Cambridge) was photographed wearing a dress made by your company. Your sales for that dress may increase greatly for a brief time, thereby unintentionally affecting your test results dramatically. You just need to be aware that’s what is causing the anomalies.
There is always something going on in the outside world, just be aware of how it could affect your business.
Wrongly assuming that a segment of the traffic accurately represents the total is known as a selection effect. It can occur when promotion acquired traffic hits your website, skewing the results.
For example, you might run a product page test using your email database traffic that results in conversions, sales, and revenue growth (much better than your site’s average), but you are testing loyal customers and so that should be expected.
Nevertheless, you are happy with the results and the conversion lift is even better than before. You declare the new variation the winner.
You then open up this winning variation landing page to all site traffic and expect it to convert at the same percentage. But, it doesn’t! Your assumption was incorrect.
Unbeknown to you, you may have bugs in your test that are producing flawed data. For example, these bugs might mean your variation isn’t displaying correctly on some browsers and devices. This is called the broken code effect.
I hope that I have helped you to understand how and why your tests are open to attack from various error causing effects. If you take control, then you’ll minimise them—and you’ll get as close as possible to eliminating their influence on your results.
It’s all down to being thorough and following a process. The more you prepare for all eventualities, the easier it is for you to maintain testing progress and get valid results.