Arunkumar Khannur's Software Testing Knowledge Center

13.10. Inquiry

Here, usability evaluators obtain information about users' likes, dislikes, needs, and understanding of the system by talking to them, observing them using the system in real work (not for the purpose of usability testing), or letting them answer questions verbally or in written form. Inquiry methods include:
  • Field Observation
  • Focus Groups
  • Interviews
  • Logging Actual Use
  • Proactive Field Study
  • Questionnaires

Field Observation
Human factors engineers go to representative users's workplace and observe them work, to understand how the users are using the system to accomplish their tasks and what kind of mental model the users have about the system. This method can be used in the test and deployment stages of the development of the product.

In order to carry out field observations, as a first step field visits shall be arranged. This involves- choosing a variety of representative users of the product, from different workplaces, industries, and backgrounds, and arrange field visits with these users. Prepare the list of questions need to be answered and data need to be collected.

Following this, field visit shall be done and as much data as possible shall be collected there. Data analysis can be done after getting back to the office. In order to collect data, during field observation effctive inquiry to be made which includes interviewing users about their jobs and the ways they use your product. Part is observation; watching people use your product in the way they normally would in the course of day-to-day life. To ensure adequate data collection it is essential to identify as many artifacts (physical objects in use at a site (notebooks, forms, reports, spaces, walls) and outcroppings (noticeable physical traits that mark or characterize the site like size of cubicles, size of whiteboards and what's written on them, uniforms written by certain castes of personnel) as possible. Then ethnographic observation means "watching peopleā€ shall be carried out.

Focus Groups
Focus group technique is a data collecting technique where about 6 to 9 users are brought together to discuss issues relating to the system. A human factors engineer plays the role of a moderator, who needs to prepare the list of issues to be discussed beforehand and seek to gather the needed information from the discussion. This can capture spontaneous user reactions and ideas that evolve in the dynamic group process.

In order to implement focus group activity, as a first step, locate representative users (typically 6 to 9 per focus group) who are willing to participate. Following this a moderator shall be selected. As a next step, a list of issues to be discussed shall be prepared and goals for the type of information to gather shall be set. Moderator shall initiate and keep the discussion on track without inhibiting the free flow of ideas and comment. Moderator shall Following this, the focused group shall write a summary of the prevailing mood and critical comments of the session, including representative quotes.

In this technique, UI engineers formulate questions about the product based on the kind of issues of interest. Then they interview representative users to ask them these questions in order to gather information desired. It is good at obtaining detailed information as well as information that can only be obtained from the interactive process between the interviewer and the user.

In an evaluation interview, an interviewer reads the questions to the user, the user replies verbally, and the interviewer records those responses. The methods of interviewing include unstructured interviewing and structured interviewing.

Unstructured interviewing methods are used during the earlier stages of usability evaluation. The objective of the investigator at this stage is to gather as much information as possible concerning the user's experience. The interviewer does not have a well-defined agenda and is not concerned with any specific aspects of the system. The primary objective is to obtain information on procedures adopted by users and on their expectations of the system.

Structured interviewing has a specific, predetermined agenda with specific questions to guide and direct the interview. Structured interviewing is more of an interrogation than unstructured interviewing, which is closer to a conversation.

After the interview, the observations shall be recorded. It is recommended to avoid making notes since it may distract the focus and flow of the interview.

Logging Actual Use
Logging involves having the computer automatically collect statistics about the detailed use of the system. Logging is usually achieved either at low-level parts of the system software, such as keyboard and mouse drivers, or by modifying the software of interest.

Typically, an interface log will contain statistics about the frequency with which each user has used each feature in the program and the frequency with which various events of interest (such as error messages) have occurred.

Statistics so collected showing the frequency of use of commands and other system features can be used to optimize frequently used features and to identify the features that are rarely used or not used. Statistics showing the frequency of various error situations and the use of online help can be used to improve the usability of future releases of the system by redesigning the features causing the most errors and most access for online help. This technique can be used at the test or deployment stages of software development.

In addition to statistical use of logging data, it is also possible to log complete transcripts of user sessions either for use in later playback or for analysis of patterns of use, such as what commands are issued next after an error situation.

A final use of logging data is to study the user's detailed use of a user interface to find usability problems that may not be apparent when observing users. The logging data can vividly show the statistics of one action (e.g. clicking on an object) from many users by integrating the logging data with the user interface.
Khannur's Book
Arunkumar Khannur, Software Testing - Techniques and Applications, Published by Pearson Publications, 2011 (ISBN:978-81-317-5836-6; Pages:341 + xxii)
Follow Khannur
Khannur's Company
ISQT Process & Consulting Services Pvt. Ltd., Bangalore, INDIA
Khannur's Software Testing Forum
 Contact Khannur
ISQT Process & Consulting Services Pvt. Ltd.
#732, 1st Floor, 12th Main,
3rd Block, Rajajinagar,
Bangalore - 560010, INDIA
Phone: +91 80 23012511
Skype: arun.isqt