Software, Web sites, and electronic products typically undergo structured quality assurance (QA) testing, since releasing a product without testing runs the risk of unhappy customers, costly returns, and damaged reputations. The need for QA testing is proportional to the complexity of the product, and modern digital products are so complex that it is impossible to anticipate all technical problems during design, so these products must be QA tested. Likewise, when designing the interaction of a complex product with a human being—who is arguably much more complex than the product—it is impossible to anticipate all the UX problems. Therefore, UX evaluation is essential. There are five primary methods of UX evaluation, (1) usability testing, (2) aesthetic testing, and (3) heuristic inspection, (4) collaborative evaluation, and (5) guideline review.

Usability testing is the evaluation of a product’s usability through direct observation of user behavior during a structured task. A typical usability test requires a small number of participants (3-10) recruited from a representative sample of actual or prospective users. For some products, users are happy to volunteer to participate, for others users are paid or offered a thank you gift. A test session may take place in the development organization’s offices or at the user’s site and typically lasts about an hour. During this time a user is asked to complete a task associated with a UX requirement. For example, when testing an e-commerce site, participants may be asked to find and purchase a specific product. Participants are asked to “speak aloud” or verbalize their thoughts as they complete the task. The test facilitator quietly observes, noting what the participant says, non-verbal signs of emotion, as well as timing the interaction and perhaps counting mouse clicks. Sometimes specialized logging software and simple Web cam are used to record all mouse and keyboard events, screen views, audio of the participants’ discussion, and video of their faces as they complete the task. This software also allows for remote testing when it is advantageous to test users from far-flung locations. After the test sessions are complete, data are analyzed, usability strengths and weakness are summarized, and possible design improvements are generated. In order to minimize development effort testing begins with a low-fidelity prototype, progresses to a high-fidelity prototype, and continues iteratively until UX requirements are met.

Aesthetic testing is the evaluation of a product’s visceral appeal through the observation of user responses to various designs. It can be scheduled independently or coordinated with usability testing. During a typical aesthetic appeal test, a user is exposed to several versions of a product and asked to rate the attractiveness of each one. Open-ended questions are also asked to get an understanding of the underlying impressions that the different designs leave. Representative Web pages, GUI screens, and photographs [53] or prototypes of physical products can be evaluated in this way. Face-to-face tests have the advantage of allowing the facilitator to follow new areas of inquiry that develop during the test; however, Web-based evaluations that allow large numbers of tests to be run economically can also be deployed.

Usability and aesthetic testing provides direct feedback on users’ experience, much like direct field observation in Phase 1, except now the response is to a proposed product. This information is extremely valuable, which makes user testing a high priority. Another way to evaluate a product is through various professional inspection methods such as heuristic inspection, collaborative evaluation, and guideline review; these approaches cannot replace user testing but can act as a valuable complement to it. Professional inspection methods do not involve any users, but rather professionals who may be UX experts, product designers, or product marketers.

Heuristic inspection is an evaluation method where a UX expert compares a prototype against predefined set of heuristics, or rules of thumb, such as “The system should always keep the users informed about what is going on through appropriate feedback.” [86, p. 134]. Typically, the inspector evaluates the product against a list of 10 or so heuristics, noting when they are met and when they are not met, and making suggestions for improvement. Heuristic inspection can be applied before user testing to eliminate obvious flaws that would otherwise waste precious user testing time. It can also identify subtle deficits that may not come up in user testing, but will never the less have a detrimental impact on long-term UX.

Collaborative evaluation is similar to a heuristic inspection but done in a group following a structured process. Collaborative evaluation is often completed by an interdisciplinary design team and can have a similar atmosphere to a brainstorming session. One advantage is it often creates high levels of buy-in by all team members, which facilitates product changes.

Guideline review is a thorough evaluation of a product’s user interface against the guidelines described in the style guide. This is similar to what an editor and proofreader do for documentation, although here the focus is on UX issues.

Work products: Usability Test Report, Aesthetic Test Report, and Heuristic Inspection Report summarize these activities’ findings and resulting recommendations. The Collaborative Evaluation Summary is a record of key action points and their rational. A Guideline Review can be a spreadsheet-like table of issues or a printout of the user interface marked up with necessary changes.

How Do The Five Phases Fit Together?

Each phase feeds into the next and iterations can occur between most phases:

  1. Business objectives for UXE leads directly into user research.
  2. User research may lead to a redefinition of business objectives if new opportunities arise, although this is unusual. User research leads into UX requirements.
  3. UX requirement analysis work products are sometime verified with users recruited during the user research phase. UX requirement analysis leads into UX design.
  4. UX design occurs in several iterations, with each iteration producing a model or prototype of increasing detail and granularity. UX design leads to UX evaluations.
  5. UX evaluation iterates back to UX design until UX requirements are met.

Now that we have considered how UX is engineered, let’s consider who uses these UX methods.

Contents, User Experience Engineering (UXE) Essentials Series