Insights, commentary and news

Conducting usability testing on websites and apps with actual users is a smart decision that can provide immense value to your designs and your business.

I’ve experimented with a variety of methodologies for conducting these studies – from expensive, high-tech, eye-tracking labs to guerilla-based tactics with random participants at my local Starbucks. For years, in-person, lab-based testing was thought to be the most reliable way to evaluate design and user interface decisions in a controlled and sterile environment.

These days, new technology services can help satisfy our ever-growing global marketing needs for user feedback. While in-person, lab-based testing gives us much more control over the environment and provides close contact with our participants, it can be costly and unrealistic. Setting up a physical lab and viewing room, along with purchasing software and recruiting diverse users who live nearby, can make usability testing seem unapproachable. Additionally, labs take the participant outside of their normal environment and technology setup, undermining the realistic tasks we so carefully crafted based on user insights.

Because of these factors, many user experience researchers in the last few years have turned to remote usability testing.

How Does Remote Testing Work?

There are two types of remote usability testing, and they are often confused with each other: moderated and unmoderated. While both allow the user to participate in the study session from their home or office, there are some major differences between the two:

• Remote, moderated testing is typically conducted via phone and/or screen share in real time, with both the participant and the moderator talking through tasks one-on-one for about an hour. This approach allows our team to directly interact with the participant, probing on interesting observations and asking follow-up questions that were not in the original script. It’s the most directly equivalent to traditional lab-based testing.
• Remote, unmoderated testing is more akin to a survey. Participants complete the testing tasks on their own time through prompts from the testing software, with no interaction from the facilitator. The facilitator receives the recordings and results as soon as the test is completed. While the results can be obtained extremely quickly, unmoderated testing doesn’t allow us to follow up or point a participant in the right direction if they begin to struggle.

Simple Solutions for Remote (Moderated and Unmoderated) Testing

It’s simple to get started with remote testing using a few simple tools, but often these tools can affect the quality of the results. Everyday technology tools for moderated testing, like GoTo Meeting®, Skype and WebEx help us connect with participants using their built-in calling and screen-share features. They provide an easy entry point into remote testing, without an investment in software or infrastructure.

However, as mentioned, these less sophisticated tools have their drawbacks:

• They often don’t address mobile device testing adequately
• Recruiting and session setup can be time-consuming
• Getting users connected to the actual sessions often results in technology hurdles
• Recording, taking notes and then reviewing the results later is clunky (and in my experience, the recordings are seldom utilized to their full potential)
• Unmoderated studies are not possible because these tools require real time participation

A New Frontier in Remote User Testing

New technology and software tools on the market today solve many of these testing issues and enable both moderated and unmoderated remote testing, depending on the researcher’s testing goals and budget. They are also frequently priced at a reasonable, per-participant rate that can be worked into the overall testing budget.

These technologies give us the best of both worlds: we can see the user’s camera, microphone and screen (both on mobile and desktop), and we can also have other observers, such as clients and account managers, participate in the sessions, taking notes in real time no matter where the participants are in the world.

In remote sessions, users perform the test on their actual personal devices, making the previously unrealistic lab-based setting suddenly extremely personal. In past tests, I’ve had users receive phone calls, notifications and use a variety of devices with different screen sizes, speeds and operating systems. These “disruptions” can often create interesting fringe cases and help us understand the user’s actual environment.

Additionally, many of these services take the hassle out of scheduling. They enable the user to choose their own time, send confirmation emails and easily connect through a link within the invitation at their designated time.

And of course, even after the tests are complete, dedicated remote tools allow us to more easily review recordings, notes, pass/fail rates and clips within a single interface following the study.

Breakdown of Testing Technologies

A few examples from the new crop of remote testing services are Lookback, Validately (now a part of UserZoom) and UserTesting.

Most of these services include unmoderated testing as well, which enables us to build the test in advance, recruit through a built-in panel and get the results back within 24-48 hours – all without the time investment required to individually moderate the study.

	Lookback	Validately	UserTesting
Moderated Testing	Yes	Yes	Yes
Unmoderated Testing	No	Yes	Yes
Lab-Based Testing	Yes; direct support	No	Yes; partial support
Unlimited Participants	Yes	Yes; if recruiting own	No
Unlimited Studies	Yes	No	No
Multi-Device	Yes	Yes	Yes
Self-Scheduling	No	Yes	Yes
Built-in Notes/Reporting Interface	Yes	Yes	Yes
Built-In Testing Panels	No	Yes	Yes; extensive
Automated Transcripts	No	No; from clips only	Yes
Live Broadcasting to Collaborators	Yes	Yes	Yes
Cost	Low	Medium	High (if testing with own panels or using with multiple clients)

Our Experience with Remote Testing

Marcus Thomas recently invested in remote usability software to provide a more flexible and agile approach to user research for our clients as we regularly evaluate new and existing websites, apps and interfaces within the agency.

In recent research sessions (we currently use Validately for testing), we’ve been able to test more often, launch tests more quickly and receive feedback much earlier in the process. We can run moderated testing sessions when the budget and need call for it, but we also have the flexibility to run simpler, unmoderated studies for projects that are smaller in scope.

So, how does remote testing compare to the in-person testing we have experience with? 

• Testing sessions are more reliable, with fewer technology-support headaches when attempting to get users to join the sessions – allowing us to focus on the actual research at hand.
• Despite not being with the user in person, we still feel we’re able to form a personal connection with participants and get honest, valuable feedback.
• We’ve found more engagement from our clients and team members because of the ease of joining and observing sessions from anywhere.
• Administrative time previously spent scheduling, recruiting, processing incentives and rescheduling has been dramatically reduced, thanks to self-service scheduling and built-in optional testing panels.
• We’re able to more quickly analyze and produce clips from testing to share with our teams and clients.
• We’ve been able to share the infrastructure of testing costs across multiple clients, saving our clients from having to purchase the software and services themselves.

In our first year using this new technology, we are confident that remote testing stacks up well (and in many cases, exceeds) traditional in-person usability testing, making it a suitable replacement.

Chris Hallahan is Senior User Experience Designer with the Web and Applications Development team at Marcus Thomas LLC. The team focuses on planning, design, development and optimization of enterprise level digital engagements for companies and brands nationwide.