MDC 2020: Watch, Listen, Learn – How to Run a (Remote) User Study
KRUTSCH returned to the Minnesota Developer’s Conference this year to talk to developers about the importance and power of a user study.
This year presented a very different experience as the three of us presented from three separate places, unable to see or hear the audience. But it did provide us the perfect opportunity to demonstrate and talk about how to run remote user studies. We’re providing a brief overview to our talk and the link to the video below. Thank you to MDC, we look forward to next year!
Watch the video or read the transcript below.
[KEN]
Good afternoon and thank you everybody for tuning in, we appreciate it. We originally outlined this before the pandemic hit, so it was 'Watch, Listen, and Learn: How to Run a User Study’ and post COVID-19, we slipped in the “Remote”. Obviously, we do a lot of remote user testing anyways so it wasn’t really that much of a change. But let’s get started here.
LESSONS FROM KEN’S FIRST USER STUDY: DITCH THE LAB
The first user test I ran was more like a criminal suspect interrogation, really, than a user study. The first one I participated in was really before video conferencing was commonplace and software in those days was something you installed off of rotating media onto a PC. So it was more convenient to set up a lab with computers with the software already installed and we’d march people in.
The first one I participated in was a company in Boulder Colorado that specialized in this and they set up a very sterile environment. It literally looked like this picture. It was a white, brightly lit room with a single table in the middle, a light, a chair, a one-way mirror behind the test subject with bleachers behind it with people observing. A proctor sitting in a chair with a clipboard and pencil judgmentally watching and checking boxes every time you did something.
I remember watching the test subject and thinking this guy is probably afraid to touch a key and every time he’d move the mouse to click on something, and he’d nervously look around and sometimes even ask, “so did I do it right?”. So after watching that, we learned a lot of lessons that had nothing to do with our product and learned more about how not to conduct a user study.
A MINI USER STUDY
So what we’re going to do today to kick this thing off, we’re going to conduct a little mini user study and we’re going to show you some of the lessons we’ve learned over the years on how to do this well. Emma will take the role of the UX researcher, and Rachelle will take the role of usability test subject. So I’ll let Emma take it away.
[Emma]
Great, so for our mini mock user study, I’m just going to go through what an introduction could look like. I’d get in touch with her over Zoom in this case, and I could say, “Rachelle, thanks for your time today. I’m going to ask you to complete a few tasks for us, you’ll have to answer a few questions. I’ll reiterate that there’s no right or wrong. We’re here to observe and learn how you interact with the application. I’m going to send you a link, ask you share your screen. The only other favor I’ll ask is that you narrate your thoughts as you click through.
So I’ll send you that link, if you’ll open and share your screen please. Okay, so we have our mock application here, and I’d just start in right away asking her some questions and then diving into some tasks.
So, Rachelle, tell me what you see on this page.
[Rachelle] Um, I can see a list of users and their email. I can see their role and then which department they belong to.
[Emma] Great. Tell me, which role does Mary Porter hold?
[Rachelle] She is a director.
[Emma] Can you tell me what permissions she has?
[Rachelle] She can record creation, editing controls and task assignment.
[Emma] Why don’t you edit her role to administrator.
[Rachelle] Um, [struggles to find where to complete this task.] I’m not sure.
[Emma] Okay, so because Rachelle’s unsure, I’d then direct her a little bit. “Why don’t you click on the tab on the right with the arrow pointing left.”
[Rachelle] Right here?
[Emma] Yes. And edit her role to administrator. Okay, now navigate to the dashboard.
[Rachelle clicks at top of screen]
[Emma] Thanks Rachelle, you’ve completed our mini user study.
———
[Emma]
This is a quick anecdote, but it really gets the point across that a user study is really asking a user to complete some tasks. We watch whether they can or cannot complete those tasks and any other questions we want to ask to further get their perceptions about the site. So that seems pretty simple, but it’s really the soft skills around it that make a user study successful and your design process successful.
WHAT IS A USER STUDY?
We’re going to take a look at what is a user study. Well, it’s what we just showed you. We have a quick definition here, but it’s what you just saw. It was a facilitator asking a participant to complete a set of tasks using a specific interface while observing their behavior, listening to feedback, and learning problems or opportunities within the design. -NNGroup
A Remote User Study
These pictures on the right are examples of what user studies can look like. A lot of time, especially remotely, they look like what you just witness with Rachelle, which is similar to the bottom left image. It’s a screen share over Zoom, WebEx, or Skype. The camera is hopefully on both participants so you can see some facial expressions and body language, and hopefully you’re recording so that you can refer back to it.
In-Person User Study
When it’s one-on-one and in person, it can look like the image on the bottom right. It’s a camera propped up on a table, aiming at a telephone, or a mobile phone I should say. The benefit of this is that you see what they’re clicking on the screen, but you also see how they interact with the device. You see how they hold the device, whether they do their thumbs hover over certain areas.
Similarly, the top right is a user study we did for [the number one health system in the nation], and that again was just a camera setup on a table. It doesn’t have to be fancy, but you really get to see the environment these women were working in, dealing with 30 years of legacy software. A lot of inefficient, and complicated workflows that they somehow mastered to complete their work, and you know, again a rather inefficient way. You get to see their surroundings and really talk to them in their “natural habitat”.
The image on the top left is an example of an in-person user study. If you recognize him, it’s Chuck Foreman. This was from a user study we did with retired athletes. You can imagine that the findings that we had from a user study with retired athletes was very different than the findings from a user study we did with Sidewalk Dog in the bottom right corner, which has more tech-savvy millennial kind of users; compared to retired athletes who have a very different comfort level with technology.
WHY A USER STUDY?
So next. Why a user study? Why do we want to do these? Really, user studies are ways to identify problems, to learn user behavior, and to discover opportunities to improve the design.
You saw this with our mini user study with Rachelle. We found some problems where she wasn’t able to complete the task. She didn’t know how to edit. Well that problem was an opportunity to improve the design, to find an interaction model that’s more usable, clearer for users to understand. We also learned a bit about her behavior. She’s not afraid to ask for help, to click around. That’s a key component as well.
It’s when you combine all these things together which we call our “eureka moments”. They’re tid bits from the user study that really lead the design in a completely different direction, and I think ways that really help differentiate the products in the end.
BEFORE YOU START YOUR INTERVIEWS
How to Conduct a User Study, Especially Remotely.
[Rachelle]
So before the interviews even start, you need your core elements of usability testing. You’ll have your facilitator. This person guides the participant through the test process. You’ll have your tasks, and these are realistic activities that the participant might actually perform in real life. And you’ll have the participant. This person is a realistic user of the product or service being studied.
Understand the Problem Domain
You need to understand the problem domain. Workshop sessions, heuristic review, interview product owners, sales, and support. Workshop sessions and heuristic reviews allow us to understand your subjects’ language. And you really want to understand the subjects’ language. For example, do they use language such as “dashboard”, “upload certificate”, or “credentials”. Those are all words you’re going to want to learn how they use and then apply them yourself.
Define Your Roles
You need to define your roles. Who interacts with the product, interface, or system? A few examples could be an admin, a director, an individual.
Create Your Task List
Then, we’re going to create your task list and questions. Your task list and questions help build the full review of the user’s experience. You’ll build your task list. You’ll want to consider common tasks per user role, specific workflows to analyze the current workflows versus ideal workflows.
And some examples for a task list. You may want to watch your user navigate to a specific page, or maybe watch them upload a photo, ask them to turn on their camera.
You also will build your question list. These are informal interview questions about the use, perceptions, and feelings. Some examples of questions could look like, “How often do you use this?” What’s your favorite part of the app? And “what do you wish you could do with this app, but currently can’t?”.
Recruit Your Participants
You obviously need to recruit your participants. In general, 6-12 users per role. A lot of people tend to think this number is too small, but we find that problems within an application tend to repeat themselves. Findings from the Nielson Norman Group support this.
If you can, recruit real customers. It’s ideal if you can get people who have used the product, but there are online tools like Userlytics that can be great for recruiting if needed. And these participants, they can be voluntary or compensated. This will really just differ for each project that you have.
Prepare the Logistics
Then You’ll Prepare the Logistics. You’ll need to collect info, contacts, availability. Tools like Airtable, Google Forms can be really great for this. You’ll choose a remote conferencing tool. Tools like Zoom or Skype make it possible to record the session, screen share, and have your camera on.
You’ll of course want to schedule your interview, send our reminder emails, do any test calls if needed. Test calls can be really great to work out any of the kinks before session starts.
DURING THE INTERVIEW: HOW TO WATCH
[Emma]
So that’s all before the interview. Now, how to “watch” during the interview.
As Ken mentioned with the story at the beginning, we really want to ditch the lab environment. User studies are most successful when the user feels comfortable and that’s not going to be in a sterile environment, away from the space where they normally interact with the app.
We really want to observe subjects one-on-one.
That’s when you’re going to get the most out of your user study. We’re big proponents of qualitative over quantitative. We think that you get better results, better feedback, better research even from that six to twelve users, even from that small subset because a lot of time their feedback does start to repeat itself, instead of expanding your sample set.
We really advise you to say no to focus groups.
Often, we find there’s one or two personalities that kind of starts to control the group think or sway opinions one way or the other. You just don’t get as great of results and observations from a focus group. And realistically, a user normally is not interacting with the app while in a group of people like that. They may be working in teams, but it’s usually just one-on-one with the screen.
Another thing to watch: watch how users interact with the app.
We talked about this when we showed some of those images about what a user study could look like. You really want to see how they click around, how they interact with whatever device they’re using, and take note of that.
And at the same time, watch the body language.
That’s going to tell you just as much as their comments do. Are they frustrated? Do they feel tense? Are they relaxed? Again, it’s all of these soft skills that wrap into those user findings that will help the user process.
HOW TO LISTEN
We should stop analyzing and note taking during an interview.
I think that the user can really tell when you’re paying attention, and you are relaxed and not frantically scribbling your notes down. That tends to make the user feel like you’re disconnected and not as engaged. And, again, it’s all about that comfort level. They’re going to start to pull away if you’re kind of splitting your attention. Hopefully, as we’ve mentioned with some of these logistics, you’ve been recording your session and you can go back and take your notes afterwards.
Similar to some other comments that we made in the “watch” section, you really want to listen to action.
Especially where the action of the user differs from what they’re saying. For example, we did a user study with a group of childcare providers that were using a software to do some record keepings. As we were watching them interact with the app, we asked them how they felt about the app. Was it usable? Were they comfortable with it? And they would say, “yeah, it feels pretty straight-forward” but then it would take them ten clicks to complete one task or they wouldn’t remember where one section was where they needed to upload a document. What their actions were telling us was very different from the “easy to use” language they were telling us.
In addition to that, you should really listen to a users’ comments.
A lot of times its some of their off-the-cuff comments that give you really clear insights into how they’re feeling. You know, we’ve had users during studies that will say things when they’re frustrated and they can’t complete a task, “I’m not good with technology” or “I just feel dumb when I use this.” That’s not great. As a product owner, you don’t want an application that makes your user feel dumb or inadequate. Those are opportunities to really improve and flip those feelings on their head and make the user feel empowered and good with technology.
You should also, again as you saw on our mini-user study, ask the user to narrate their thoughts so you can listen to their thought process as they’re clicking around.
Listen to their tone.
Similar to body language, are they getting frustrated, are they relaxed, are they getting excited about something? Tone can tell you a lot. Listening is really important because a good UX researcher doesn’t like to listen to themselves talk, they really let the user do all the talking. That’s how you’ll learn, in addition to a few points we’ll talk about next.
HOW TO LEARN
Get to know users’ background.
You can really learn a lot when you learn about other people’s experiences. So, when you’re talking to a user, we like to ask about their background. For example, if you have a user that maybe had a background doing some data entry in their previous job, that will give you some insight into why they’re so good at the data entry portion of an application, compared to other parts of the app. It really changes the lens through which you’re viewing their comfort level.
And you can also let the user get off task.
This is when we find some of the best little “nuggets” from our user study is when we don’t stick to the script. It’s okay if they get on tangent, or they’re pulling you in a different direction, that’s all good. It shows that they’re comfortable with you, it shows that there’s something they’re excited about, or that feels natural to them. Those are all things you want to pick up on and hopefully try to capitalize on in your design process as you find those opportunities to improve.
Additionally, you learn a lot when you observe their surroundings.
Like some of those images we saw of what a user study could look like, you can see what their surroundings are like. Do they have to screens up? Is it hectic? Do they have kids running around and screaming in the background? Is it isolated? Are they flipping between devices? All of these things really go into some of the best tidbits from a user study.
Keep it professional, but informal.
But overall, you learn the most when you keep a user study relaxed and informal. That’s when people start to open up, that’s when you start to open up and connect with the user and that’s when you’re going to learn the most.
AFTER THE INTERVIEW
Collect Your Observations
[Rachelle]
Now you’re going to collect your observations. Now it’s time to review your recording. Consider making annotated notes with time stamps. Consider taking it a step further and grouping observations by type or category. You could use a Toole like Airtable.
Here’s a quick example of some observations here with timestamp. You can see the user’s name; we’ve highlighted some quick notes. It’s a rough outline for us for when we go to put together our highlight reel. You could also take it a step further. Here’s an example using Airtable. You could break down your annotations and sort by a specific focus. Maybe you want a lot of detail on what your users were thinking, or how they felt throughout the process.
Some questions that guide observations:
· What did users say versus what did they do?
· Where did they struggle? What came easily? Shortcuts? Long Form?
· Any improvised workflows?
· Note their setup. Desktop? Tablet? Dual Screen?
· The observation that matters most: Could they complete the task?
Build a Summary of Observations and Highlight Reel
The point of all of this is to show the product owner what we’ve learned. We do this with a highlight reel. Summary of observations serve as roadmaps for developing feature set and workflow. You’ll put together a highlight reel with potent clips from user studies. You’ll build your clips around an observation. For example, maybe people were struggling with a key observation or task. During your user study, did you ask participants to update their contact info and most of them failed? That’s something you’d want to highlight.
A highlight reel forms buy-in from clients and team members. Watching all of the users struggle gets everyone on the same page.
Example of a Summary Observation:
"Overall, users have an overwhelming support of the system and its goal—so much so the they’re willing to learn/live with the system’s quirks. However, even though they say they love the system, they don’t use it for its core purpose like finding trainings or managing employee records. They often create their own systems.”
An Example of a Highlight Reel
We have an example of what all of this could look like. We see we have our observation of Manual Systems for Tracking Hours. We can see that our camera and our user’s camera is on. We’re able to have screen share enabled so we can see what they’re doing. It’s really important to have your user’s name and their role.
WHY USER STUDIES WORK
[Ken]
Save money in the long run.
Well, they work for really simple, and obvious reasons. First of all, and most importantly, you save money in the long run. If you do a user study correctly, you tend to solve the right problem. My experience has been that when you blow off that step, and don't look at any customer research, you can end up solving the wrong problem.
It builds user loyalty.
If you build a product that fits someone’s needs, that tends to create positive brand association and therefore user loyalty.
It makes the design process more productive. A lot more productive.
When you have recorded observations to back design decisions and directions, it tends to take a lot of the opinion driven arguments out of the design process. I like to tell people, the phrase is “I think” is replaced with “We’ve observed”. What we mean by that is, you guys are developers in the audience, right.
We’ve all been in meetings where we’re talking about a design decision for a piece of software that you’re building. When it comes to UI design, I’m sure you’ve all seen it, everyone is an expert. And everyone has a lot of opinions. So you hear lots of, “I think we should do this.” “I think users won’t like that.” “I think users won’t figure this out.” “I think users…” “I think, I think, I think,” And when you can then replace that whole conversation with, “We’ve studied a group of users who are from our customer base and we’ve observed that they are asking for this feature or they don’t use that feature or they cannot figure this out, you know etc., etc., etc. And it’s really the most important reason to do a user study. To drive the product design.
REBUTTAL POINTS
Well, we hear these a lot. And maybe you guys hear these too. The kinds of things that you’ll hear from engineering management, product management, maybe the executives that work at a company that are fielding a product:
“We know our users.”
That’s the most common one. “We know our users. Trust me. We know everything there is to know about our users. Trust me, I’ve been doing this for 20 million years and I’ve seen it all. We know our users.” OK. But every time we run a user test, and I mean EVERY time, we discover something important about the customer base that none of the stakeholders knew about their users prior to the study, every single time.
“We don’t have time.”
Really. So, you don’t have time to take a couple of weeks and just check that you’re building the right thing before you embark on a yearlong development project. OK
“We don’t have the budget.”
Well, see above. Again, you’re talking about spending a lot of money, you guys in the audience build software for a living so you know how much this stuff costs to build and what does it cost to build the wrong product?
Some of our “Eureka Moments” that address some of our previous points. I could make a whole hour out of a list of this stuff. Here’s a list of some that came to mind as we were putting this presentation together.
OUR EUREKA MOMENTS
Unboxing Not to Plan: Device Mix Up
There was one example where we were doing a user study for a consumer electronics company that was building products or consumers of the home and we actually went out to people’s homes and we observed the trying to set them up. And the subject mixes up similar looking devices after removing them from the packaging. The product was two boxes, two little boxes that did completely different things, but they looked exactly the same. So, what did the customer do? The opened the package, they take everything out and throw it on the table. Then they grab maybe some directions that say, “take Product A and begin setup as follows”. And they would pick up Product B and have no idea they had Product B, and you know chaos ensues. Something that the engineers in the company, you know it never occurred to them.
Slow Response Times Negatively Affect User Expectations
Second one. Subject starts a task and then gets up to get more coffee. So, we were doing a user study for a storage area networking company that had a user interface for a big storage area network. Product responsiveness in real world configurations was so poor that users would click on something and expect to wait minutes for a simple task to complete. So, I had asked him to execute a task and he goes, “clickety-click”, gets up out of his chair to walk out of the room. I asked, “where are you going?” And he said, “I’m going to get some coffee, this is going to take forever. You want to come?”
System Too Complicated, Users Find Workarounds for Core Tasks
The next one. This is real too. Subject glares at you with a furrowed brow and tells you “This f—ing system makes me feel so stupid.” Blah, blah, blah. The product was so difficult to use, real customers were actually doing the real work in Excel and then pasting the results back in. I mean this was a data entry application and they were using excel to do all the same thing and then they had it the way they liked it, they’d copy and paste and go “Done”. That was also something that the client never knew about.
I think that’s it.
Thank you. Thanks for your attention, we’d love to take questions. Fire away.
Emma Aversa is a visual designer at KRUTSCH, with a background in architectural design. She excels at digital design and communication executed through simple, function-forward designs.
Rachelle Abernathy is a graphic designer at KRUTSCH whose work focuses on integrating beauty into the pragmatic function of our everyday life. She’s a creative thinker, a perceptive collaborator, and an enthusiastic worker.
She believes in fostering a collaborative team dynamic to best serve our customers, and is committed to finding creative and efficient solutions to design and business challenges.
