A Case Study with Card Sorting

Jan 25
5

Businesses commonly use internal language to construct web sites, which may not align to the language of the user. Card sorting provides an effective way of generating user feedback by revealing information about the user’s mental model.

Process

In a typical card sort, users are given cards containing different site content and asked to organize the cards into groups as they see appropriate. Users are sometimes asked to create a name for each category they have created (known as an open card sort), or they are provided preexisting category names by the design team and asked to place the cards into one of the predefined categories (a closed card sort).

Online Card Sorting

Online card sorting enables remote testing, which provides a wider sample of participants than the physical restrictions manual sorting allows. While there is no significant difference between the results rendered from manual versus electronic card sorting, the online process often takes longer for a first time user to complete (Harper and Van Duyne, 2002). Additionally, the computerized nature of electronic card sorting affords quick generation of results, which leads to a faster analysis.

Design For Use Case Study

Design For Use employed the electronic card sorting technique to gather information about users’ mental models of a government retirement system web site. The goal of the exercise was to determine how users would categorize forty cards from the current retirement site that are ambiguous in their placement, or exist as outliers to the content with which they are currently grouped. For example, there was uncertainty about where to group “Legislative Updates”: does it belong with “News” or with other information pertaining to state laws? Design For Use used OptimalSort, a tool created by Optimal Workshop, to facilitate the card sort.

Methodology

Participants

Design For Use recruited a user sample populated by present employees, retirees, and former employees, as well as participants of varying skill levels. Considering the large number of people who access the government retirement site, the participant sample was wide enough to reveal information about each user group.

Protocol

  • Participants accessed the online OptimalSort survey by providing their email address.
  • A questionnaire was administered to determine their type of internet access, the frequency with which they used certain web sites, and the comfort they have navigating web sites.
  • Participants were shown instructions on how to complete the card sort, with the help of screenshots from the application for each step.
  • After viewing the instructions, participants were then able to begin the exercise.
  • Participants completed the card sort at their leisure, and had access to the instructions throughout the activity.
  • When finished, participants logged out of the system and their results were saved for the researchers to access at a later time.

Participation

Of the 57 participants in the card sorting exercise:

  • 31 (54%) sorted each card and labeled each category
  • 2 (3%) sorted each card and labeled some categories
  • 10 (18%) sorted the cards but did not provide any category labels
  • 14 (25%) logged into the exercise but did not sort or label any cards

The participation results indicate that 76% of respondents engaged in the card sorting exercise in some way, even if they did not complete the activity. Considering the goal of the card sort was to learn about user’s mental models for forty terms that were scattered throughout the original site, the amount of feedback is appropriate and the card sort was successful.

Data Analysis

Finally, the data retrieval and analysis of user’s feedback also commonly proves a challenge in card sorting, even despite the electronic nature of the information. Consequently, the raw data downloaded from OptimalSort initially appeared complex and incoherent in an excel spreadsheet. However, after utilizing a template created by Joe Lamantia and supported by OptimalSort, the results took shape after some minor data entry and specific category normalization. By determining the frequency with which each card was placed in a given category, the researchers could establish to which category the card belongs.

Raw Data from OpitmalSort

Data Analyzed with Lamantia's Template

Normalizing Category Labels

Accordingly, one of the most challenging aspects of the data analysis in an open card sort is the normalization of user-created categories. While closed card sorts provide the users with the categories for the cards, open card sorts allow users to create their own category names.

When normalizing the category labels, the researcher is ultimately completing a card sort by dividing the user-created categories into groups. If the researcher tries to maintain the complexity of the user’s feedback by keeping a large number of categories, the data is too dispersed to be meaningful in creation of labels for the web site.

For example, users created numerous labels for the category pertaining to health insurance and medical benefits, which is labeled “Insurance” on the current retirement site:

  • Benefits
  • Medical
  • Medical Benefits
  • Insurance
  • Healthcare
  • ERS Services

Each of the categories above contained many of the same cards, and to make effective use of the data, the researcher needed to condense the category names. The ultimate category label became “Medical Benefits” based on the popularity of those labels and extent to which those names overlapped.

Top Findings

Replace ‘Insurance’ Label with ‘Medical Benefits’

While the current site uses the word “Insurance” to label a page about employee medical benefits, the card sort revealed that a majority of users of this system employ the term “Medical” or “Benefits” to describe the features found on this page. Considering the most commonly searched terms on the web site are found within the current “Insurance” page, users are having trouble locating that information on their own, possibly due to the fact that the “Insurance” label does not intuitively match the information on the page.

Create Static Pages for Dispersed Content

There is no definitive page for “Forms”, “New Employees”, or “Tools” on the current web site, as these resources appear throughout the web site. For example, medical claim forms appear on the Insurance page, retirement forms are found on the Retirement page, and new employment forms are found on the Publications page. Considering the users created specific categories for each of these resources, it might be helpful to create pages where all forms, information for new employees and tools can be found.

Conclusion

Design For Use conducted a worthwhile and compelling card sort to reveal the users’ mental models of the government retirement site. Specifically, by providing a pair of objective eyes, Design For Use was able to determine that the language of the web site did not always align to the language of the user. Oftentimes, businesses work too closely with the needs of the customer to recognize the disconnect between the internal structure and external function of the web site.

The card sorting exercise is one step in the larger ADePT (Analysis, Design, Prototype, Testing) process that Design For Use implements for web site improvement. Card sorting is part of the Analysis stage, which identifies the user’s needs and how they relate to your product. Specifically, the Analysis phase helps differentiate between what users say they want and what they actually want through a series of interviews, focus groups, web logs, and card sorts. The Analysis stage in turn enables an effective and realistic Design phase to emerge.

Contact Design For Use

To ensure your web site is meeting the needs of your users, Design For Use can apply the ADePT process to reveal helpful information about your users’ mental models.

Contact Design For Use to discover the best way to enhance your customer’s experience and help you accomplish more with your web site.

Posted by: David Richard
No Comment
  1. Todd Follansbee

    Mar 3, 2010 - 04:27 PM

    It would be interesting to see the results for the site although I expect that they may be proprietary. There seems to be quite a bit of inconclusive data from the sounds of it, or at least the guarantee of meaningful data is dependent upon addressing a number of challenging variables (the old breadth vs. depth discussion). I am intrigued though and will go check out OptimalSort.

  1. Steve

    Oct 10, 2013 - 02:00 PM

    Here’s another alternative tool: http://www.usabilitest.com/card-sorting

Leave a Comment