Why laboratory-based user testing is better than remote

By GarretCunningham

06.09.2016

  • Lab-based user testing gives optimisers the opportunity to select participants that match a website’s target audience
  • It provides many additional layers of data you can only capture in a controlled and comfortable lab environment
  • The entire testing experience in a lab is interactive and that reveals additional insights you cannot get from remote testing

User testing is extremely powerful and that’s why we class it as the star player in the conversion optimisation team. It’s the best way to get meaningful insights about your website performance.

When you user-test websites you have two options: remote and lab-based. Both can be moderated, if you want to maximise control and depth of your studies—although moderating remote testing can be very awkward because of the additional tech you have to use to actually sit in on the test.

So, here’s the case for doing your user-testing in a moderated, laboratory setting.

A well-designed usability testing laboratory enables maximum, meaningful data capture when testing. And, the testing can be for both qualitative and quantitative purposes.

Importantly, lab-based testing gives you much more control over the selection of test participants. For example, you can screen all candidates for suitability for the tests, ensuring that they are truly representative of the target audience and are, thus, motivated to perform the test as an actual customer.

Access additional important layers of information
Lab tests enable a wealth of information to be collected from eye tracking, click and scroll mapping (it is even possible to run brain scanning, heart rate monitoring, and galvanic skin response testing if deemed appropriate), as well as verbal responses. And, you can capture test participant expressions as they perform the test.

In a lab, therefore, you’ll get to measure a wide range of things from eye movement and time to fixation, mouse and scrolling movements, and any “physical” functionality issues right through to the level of satisfaction that the test participants have with the web site being tested.

Plus, not only does the test moderator see what is going on, the test can be further watched from an observation room. From there, other observers can feedback to the moderator about what they are seeing.

The data collection capabilities of the lab are, therefore, far reaching, and reporting is comprehensive and the insights are extremely valuable.

 

Remote testing doesn’t deliver depth of data

Some may argue that testing in a lab is expensive and is artificial when compared with remote tests performed by a participant in their own home. But, that doesn’t really cut it as a reason for choosing remote testers; it’s a test—so the situation is artificial anyway and a higher number of remote tests are needed to gather the required data resulting in higher costs.

A remote test in a participant’s home doesn’t really emulate a voluntary, self-motivated online shopping experience. Whereas, in the lab, there aren’t any distractions and experts are at hand to make the participant feel at ease and help when problems arise.

Variable quality of remote test participants

In a remote, unmoderated usability test, your participants complete specific, pre-defined tasks—and they do it solo. Their actions on-screen and the accompanying audio are recorded by their own computer, tablet or smartphone. So, you are reliant on the quality of the recordings you’re sent to watch and you have to analyse and draw your conclusions based on what is delivered to you. Which might prove frustrating if you get a number of “professional testers” commenting on how you ought to look at your branding and change your fonts, etc., instead of just getting on with the test and commentating on the task.

Furthermore, you have limited control over who your test participants are and whether they truly reflect your target market. So, there is a real issue of knowing exactly who are the test participants and whether they are approaching their tasks as a job—as mentioned earlier there really isn’t any intrinsic motivation going on to perform the test as a real shopping exercise. Plus, your test may be just one of many they’ve done that week.

But, you might be fine with the quality of the recordings if all you want to do is see how they performed small tasks like going through a checkout, signing up for a newsletter and so on.

Limitations of remote testing

Can work well for small tasks but limited because:

  • Limited options when selecting suitable users—screening questions are also limited by accurate responses
  • They aren’t moderated
  • Participants will include not genuinely “in market” users
  • They have no real desire/motivation

Users were not actively ‘in market’ so product finding tasks had no genuine motivation or thought into the process (generally people select the first item they see)

  • This does not truly test filters, search, etc.
  • It also renders comments on product description/pages less useful as there is nothing specific they are checking against

But it can pick up some useful general bugs and difficulties that support heuristic analysis and for functionality checking, etc.

Lab results = best results
So, if you are looking for in depth user testing insights, our recommendation is to do it in the lab. If all you want is “quick and dirty” tests done on small tasks or rapid feedback to design ideas, then remote tests can work well—for example five second tests can be very effectively run remotely. But, don’t expect remote testing to reveal anything other than something works or doesn’t work. You’ll need to dig a lot deeper to find out why before looking for solutions. Use a lab and you’ll get a much fuller picture with easy access to quality data that you can really use.

Read on to…Dark patterns: Evil or genius?

GarretCunningham
Hi, I’m Garret. I’m a digital marketing specialist with over 10 year experience in digital media helping clients to manage multi-channel e-commerce strategies in both domestic and international markets. Over the years I have led strategy and planning across all digital media channels including search and display, and also expanded this out to cover offline channels including TV, press and OOH. I believe the next level of growth for many businesses will come from not just knowing what their customers do on site but investing in knowing more about why they do it.

Our Clients