Arunkumar Khannur's Software Testing Knowledge Center

13.9. Usability Testing or Evaluation

In usability testing or evaluation, usability properties are assessed by observing how the system, or a prototype of the system, is actually used by some representative of real users performing real tasks. The evaluators use the results of such testing to see how the user interface supports the users to do their tasks. Testing methods include the following:
  • Thinking Aloud Protocol
  • Question-asking Protocol
  • Co-discovery Learning
  • Coaching Method
  • Performance Measurement
  • Remote Testing
  • Retrospective Testing
  • Shadowing Method
  • Teaching Method

Thinking Aloud Protocol
During the course of a usability test, the test users are asked to verbalize their thoughts, feelings, and opinions while interacting with the system. It is very useful in capturing a wide range of cognitive activities. Think aloud protocol requires one usability expert and a minimum of 4 users who should be observed - the more users that can be observed, the better the results. Different Users will have different problems.

Two variations of thinking-aloud protocol technique are:
  • Critical response - This requires the user to be vocal only during the execution of certain predetermined subtasks.
  • Periodic report - This is used when the task is complex and makes it difficult for users to think aloud while performing the task at the same time. The user, therefore, verbalizes at predetermined intervals of time and describes what he/she is currently trying to achieve. The length of the interval depends upon the complexity of the task. This technique is very time consuming, so it is recommended for subdivisions of a task.

Thinking aloud allows testers to understand how the user approaches the interface and what considerations the user keeps in mind when using the interface. If the user expresses that the sequence of steps dictated by the product to accomplish their task goal is different from what they expected, perhaps the interface is convoluted.

The terminology the user uses to express an idea or function should be incorporated into the product design or at least its documentation.

In order to carry out the tasks related to think aloud protocol technique, all the information or resources a user might need (tangible or intangible) to complete the task shall be listed out. Examples include valid login, business policies, credit card, file names etc. All these need to be available to the user during the usability test. Also, conditions and prerequisites are to be in place at the start of the task. Example – a database with a built in error to see how the user handles it.

Following this, scenario shall be written down briefing the steps you expect the user to go through. Example – Open order form. Fill out form. Submit form. Also, any options that user may have to choose have to be include. Example – Search OR Navigate

Following this, the interface being presented shall be described. Usability expert explains goals of the session to the test users and also, explicitly mentions about in-room observers and/or videotaping. The expert brings the professionalism by making the people to focus on testing the user face and not to be sensitive towards in-room observers and/or videotaping. The experts explain how to interact with the prototype and clarify tasks if confusing. Experts shall encourage test users to thinking aloud and asking questions.

With this preparation, test user start testing user interfaces. When testers complete every task they are asked to furnish details that include: a brief descriptive name and number for the task, status of the task that will include- What will users have accomplished when they have finished the task? How will they know the task is complete?

While carrying out these tasks, time taken by system to process shall be ignored. However focus on time spent entering data and clicking buttons shall be kept track. Some tasks require time for creative effort so allow time for that. Use this estimate to help you decide how many tasks will fill your allocated time. Multiply by a factor of at least 3.

Usability expert shall sit quietly and observing the test. He shall take notes. If usability expert feel he understood the issue, he sahll ask the the user to end the task and begin the next. All questions shall be deferred till the end of each task and the end of the test.

Usability expert shall make the user to explore more an area of difficulty, rather than trying to get through all the tasks. Users may rush one task if they feel they are spending too much time on others. During carrying out tasks, if the user got stuck or went down the wrong path then usability expert shall analyze the problem and extend proper support so that user continue.

Think aloud protocol technique can take up a lot of time and provide limited results. Focus on trying to understand the problem. Come up with solution later with your design team.

Question-asking Protocol
During a usability test, besides letting the test users to verbalize their thoughts as in the thinking aloud protocol, the testers prompt them by asking direct questions about the product. In Question Asking Protocol, 1 expert and at least 4 users are required.

Question Asking Protocol is used in order to understand the users mental model of the system and the tasks, and where they have trouble in understanding and using the system. This is a more natural way than the thinking-aloud method in letting the test user to verbalize their thoughts.

In order to carry out question-asking protocol technique, first the test users shall be provided with the product to be tested (or a prototype) and a set of tasks to perform. Then experts ask the participants to perform the tasks using the product, and explain what they're thinking about while working with the product's interface. Also, experts ask test users pointed, direct questions about the product. Responses from test users, either in terms of the product being tested or in other products from their past experience, will provide insights into their mental model of the product.

Co-discovery Learning
Co-discovery Learning (CL) is an adaptation of the most commonly used type of user testing - think aloud protocol (TA). In CL, users are grouped in pairs. It is preferable to pair two users who know each other into one group so that they won't feel uncomfortable working together. Then product under test (or a prototype of its interface) and a scenario of tasks to perform are provided to test users. Now, test users start to perform the tasks using the product, and talk aloud naturally to each other whilst working together to accomplish a common goal using the product. CL technique facilitates talk aloud among users who are grouped in pairs and there by erroneous assumptions about the system can be noted. This technique can be used in development stages: design, code, test, and deployment.

Co-discovery Learning is more realistic than a single user scenario, as people in work environments often work in teams. The users often find it easier and more natural to vocalize thoughts with a colleague present. The evaluators can also quantify the time taken for various tasks, the number of tasks competed correctly, the error frequency, numbers of times the users accessed the help system etc. From this they can make more qualitative judgments such as the success or lack of the entire system; system sub-components; help system; effort required to achieve a particular result; quality of interface. In summary, this technique helps to test effectiveness and satisfaction related issues. However efficiency related issues are not covered.

For Co-discovery Learning (CL) technique we require 1 usability expert, 1 camera operator and a minimum of 6 users who should be observed in the exercise.. Different Users will have different problems. The number of users must be even since pairs of users are observed.

In order to perform Co-discovery Learning, first step is to pair the test users into groups of two. It is preferable to pair two users who know each other into one group so that they won't feel uncomfortable working together. the test users shall be provided with the product to be tested (or a prototype) and a set of tasks to perform. Then test users perform the tasks using the product, and explain what they're thinking about while working with the product's interface. As the Co-discovery Learning (CL) technique progresses, test users shall be made to gradually move from simple tasks to more difficult ones. During the testing, test users can help each other in the same manner they would if they were working together to accomplish a common goal using the product. At the end, test users are asked by the expert to record their findings in an evaluation form or memory test. Also, experts evaluate, analyze videotapes and record their findings. Findings recorded are reviewed by designed and are incorporated into the prototype which is again retested.

Co-discovery Learning (CL) technique has certain limitations. This method is quite time intensive. Co-discovery Learning may not uncover all errors as users are doing only a subset of the full amount of tasks. Users may complete tasks in spite of problems or errors encountered either through persistence or learning. Some errors may be impossible to find by this method, such as poor window or navigation structure which is still usable but not optimum. User testing only uncovers symptoms of problems, not the cause – this must still be inferred by an expert. Using two people at a time means that it can be harder to find a large enough representative group to conduct the tests.

Coaching Method
Coaching Method is a technique in usability testing that involves 1-4 participants and 1 usability expert coach. In this technique, Instead of disallowing questions, the participants are encouraged to ask any system-related questions of an expert coach who will respond to the best of his or her ability with appropriate instruction. Usually the tester serves as the coach. One variant of the method involves a separate expert user serving as the coach, while the tester observes both the interaction between the participant and the system, and the interaction between the participant and the coach.

The purpose of this technique is to discover the information needs of users. By hearing typical user questions, problems are identified and help to design better training and documentation, as well as possibly redesign the interface to avoid the need for the questions. When an expert user is used as the coach, the expert user's mental model of the system can also be analyzed by the tester. In addition, since users learn more quickly, relatively expert performance can be studied.

The tester can also control the answers to certain predetermined information. In an extensive series of experiments, one could vary the coach's answers in order to learn what types of answers helped users the most. But this requires skilled and careful coaches since they need to compose answers on the fly to unpredictable user questions.

By using this technique, we can uncover usability related issues related to effectiveness, efficiency, and satisfaction. This test can not be conducted remotely and only can get qualitative data.
Khannur's Book
Arunkumar Khannur, Software Testing - Techniques and Applications, Published by Pearson Publications, 2011 (ISBN:978-81-317-5836-6; Pages:341 + xxii)
Follow Khannur
Khannur's Company
ISQT Process & Consulting Services Pvt. Ltd., Bangalore, INDIA
Khannur's Software Testing Forum
 Contact Khannur
ISQT Process & Consulting Services Pvt. Ltd.
#732, 1st Floor, 12th Main,
3rd Block, Rajajinagar,
Bangalore - 560010, INDIA
Phone: +91 80 23012511
Skype: arun.isqt