Old Habits Die Hard
One of the most insightful panels at South by South West was a presentation by design guru Steve Krug. Specifically, Krug touched on the importance of performing usability tests quickly and often during the design process. Some of his ideas seemed contrary to standard usability testing practices: recruiting loosely around the core user group, testing only 3-4 users at a time, conducting tests weekly, bypassing a comprehensive usability report for fewer more directed findings, and conducting tests in-house, rather than reserving a space at a neutral third-party market research firm.
Krug’s recommendations were understandably hard to swallow, especially considering many of the applications we work with require significant background knowledge and context to engage with the system meaningfully. Recruiting “loosely” outside the primary user group would return useless results in all but the most basic of online tasks. Additionally, many of Design For Use’s clients are out of state, thereby negating any possibility for weekly usability tests.
However, his insights have considerable merit in providing new options for usability testing. The most significant take away from Krug’s presentation regards the idea of conducting tests in-house.
Save Money, Save Time
Reserving space at a market research firm is a costly endeavor. Moreover, the difference in price between half-day and full-day rental typically compels our clients to reserve the facilities for a full-day (it’s a bit of a scam on the part of the market research facilities but that another blog post entirely). While 5 participants generally provide a reasonable threshold for identifying key design flaws, if you’re paying for a full day, you’re likely to test 6-8 participants. Krug suggests testing only 3-4 users at a time; if testing is conducted at the client’s office, there are no rental fees to consider, and testing 4 participants become cost-effective.
A Captive Audience
Additionally, conducting the test in-house is a convenient way to ensure that stakeholders can be present for the sessions. One of Krug’s major points was to move away from the two-sided mirror at traditional facilities and instead, bring the test to the stakeholders. His reasoning is simple: if a company is spending thousands of dollars on a usability test, they will be more likely to view the sessions if they take place just down the hall from their office. Once stakeholders view one session, they may even be inclined to rearrange their busy schedules to see additional test participants. Or they might call in Mike from development to witness what’s happening down the hall.
Some design mistakes are imperceptible to the developer, marketer, project manager, copywriter, or even the usability consultant. Only the user can show the true flaws in the design, or validate a need for change. However, the usability findings typically mean nothing if the decision makers and stakeholders aren’t able to witness the behavior of the user directly.
I’ll Believe it When I See It
Occasionally, a few junior team members are present to view the test sessions off-site, but often the primary decision makers are unable to break away from their schedule to see a single session. Since the team leaders didn’t witness the user behavior, the findings are often lost on them and go sometimes go unused. (Even burning a DVD of the sessions does little to motivate change, although it will make a nice coaster for a lucky stakeholder).
After screening and recruiting just the right users, writing specific test plans, conducting the tests, analyzing the results, creating reports, and presenting the findings, the key decision makers are much more likely to execute the findings of a study if they are present for the test.
On the other hand, Design For Use has worked with teams who conduct most of their web planning by committee: always seeking input from various departments before a decision is finalized. In one case, the major decision maker was present for nearly all of the testing sessions, and agreed with the top findings, after witnessing the user’s behavior first hand.
Unfortunately, when the lead stakeholder went back to the team with our report, some of the top findings were dismissed. Despite the overwhelming data and the support of a key decision maker, some recommendations were vetoed by the rest of the team, many of whom were not present for any sessions of the tests!
When Krug speaks to the advantages of testing in a convenient location for stakeholders, experience tells us he’s right (or at least he’s on to something). There is no better way to understand the results of a usability test than to watch the user’s behavior with your own eyes.
The times when Design For Use has been most successful in translating test findings into reality occur when a range of stakeholders are able to observe the test participants. While we’ve seen a mix of participation from decision makers at off-site testing tests, we’ve consistently had more luck with investing team members in testing when we set up shop close to home.
What About Neutrality?
However, there is something to be said for the objectivity of a third-party testing site. No user wants to come into the headquarters of a company and point out all the flaws in their website or prototype. Conventional testing wisdom tells us that if users come to a neutral space, they may be more relaxed and honest.
If users are encouraged to be critical and honest for the sake of improving the system; if they’re invited to share their thoughts for the benefit of other users, it is possible to compensate for the neutrality offered by a third-party location (not to mention the cost incurred by renting a facility for several days).
The Gospel According to Krug
While Krug’s advice should be taken with a grain of salt, depending on the needs of users and clients alike, he offers a fresh perspective on how to capture the attention of decision makers, gain the approval of stakeholders, and save money on facility costs to boot.