Arunkumar Khannur's Software Testing Knowledge Center

14.4. Steps from Error Reporting to Resolution

Following are the steps from error reporting to resolution:
  • Execute Test Case
  • Compare Expected Result with Actual Report, and in case of difference report error.
  • Analyze reported error to check whether it is a bug or defect and also, for its impact on cost-schedule-effort. Also, whether to accept or reject or keep it pending.
  • If accepted, allocate to developer with schedule and effort constraint. This is being done using test measurements. In brief, thumb rule being used shall be higher severity lesser shall be the time to fix. Provide access to Configuration Library with appropriate access level and privileges.
  • Developer shall take appropriate action on the error that involves identifying affected parts, localization of error, fixing the error, and report status as ‘fixed”.
  • Tester shall build the system and perform regression testing. Report the status as Closed or reopen.
  • Update relevant documents.
  • Baseline the modified documents and also code.
  • Remove access to Configuration Library.
  • Release new version of system.
Following sections describe each of these steps.

14.4.1. Executing Different Levels of Tests
Under the directions and constraints of Master Plan tests at component/unit level, integration level, and system level are planned and executed. In order to facilitate these different levels of tests and execute them, one has to define and use Unit Test Plan, Integration Test Plan, and System Test Plan. The purpose, objectives, input resources, initiation and execution, skill requirements etc of each of these tests are different. Studies and data indicate that unit test and code walkthroughs are the most effective means of reducing rework time since they ensure better code and removal of errors at the earlier phases thus by restricting error injection rate.

14.4.2. Reporting Error
Testers report errors when they find variation of software product or its components with specifications and standards. In this step, tester executes Test Case/ Test Script observes outcome. He then compares outcome with Expected Result. If the outcome is an Expected Result, then he reports as “Pass” in Actual Result column, otherwise reports as “Fail” “ in Actual Result column.

14.4.3. Error Classification
These errors are classified into: Bugs and Defects. This classification is being done using appropriate criteria that involves:
  • Implication of error on system operation: System crash or system is operational but does not behave as expected.
  • Severity of Error: Critical/ Major/ Minor/ Cosmetic.
  • Impact of Damage on cost and/or human life.

Bugs are acceptable and unavoidable in any software or system. When bug appears system remains operational however results may not be as per expectations. There can be major functional misbehavior, or minor problems like truncation or round of errors, or cosmetic errors like color, navigational, and alignment errors. Bugs do not have major impact or damage on cost and/or human life. Bugs may result in end user dissatisfaction.

Defects are unacceptable and disastrous in any software or system. Defects are showstoppers or make system un-operational or system crash. Defects may appear because of missing functionality or wrong implementation or providing extra feature. Defects are associated with heavy damage of cost or loss of human life. Defect results in client dissatisfaction.

Some organizations use different criteria in defining bugs and defects. In these organizations, errors reported in on-going or in-progress activity are defined as bugs. Errors that appeared in earlier phases or base-lined work products are referred to as defects. One can overcome this problem in the organization in arriving at working definitions being used in the organization so as to ensure uniformity and consistency in usage of terms.

We can find more details and next level of categorization of errors in Defect Prevention and Management Part.

Error Classification: Severity-wise
Each company will be having its own criteria. For example, error can be of:
  • Severity 1 or Critical- if it makes System inoperable;
  • Severity 2 or Major- in case major functions disabled or incorrect;
  • Severity 3 or minor - if minor functions disabled or incorrect; or
  • Severity 4 or cosmetic - in case of superficial error.
Based on severity of defect, we can decide on time within which defect shall be fixed. Higher the severity better shall be the response time.

Error Classification: Origin/ Phase-wise
Based on its source of origin, errors origin or phase is being identified. For example, we may find and report error while reviewing design phase work products however analysis may indicate that this error has origin in requirements. Under such scenario, we record origin or phase of error as requirements. This information of error origin or phase of error will help in identifying phases that may require improvement. In addition to phases, error origins may include documentation and bad fixes. Also, phase-wise containment will allow us to arrive at organizational norms. Using these organizational norms, we can define number of errors to be reported in each phase of the project.

Error Classification: Class-wise
Based on Class-wise Categorization, errors may be classified into:
  • Missing: Information that is specified in the requirements or standard, but is not present in the document, or
  • Wrong: Information that is specified in the requirements or standards and is present in the document, but the information is incorrect; or
  • Extra: Information that is not specified in the requirements or standards but is present in the document
This information will allow us to understand how good we are in defining scope of the project and also, requirements traceability aspects.

Error Classification: Cause-wise
Few causes for error to appear are:
  • Inadequate or non-existence procedures or documentation or standards
  • Inadequate input from Client
  • Non-compliance with procedure
  • Poor Estimation and/or Scheduling
  • Lack of Training
  • Inadequate working conditions
  • Lack of Man-power

The information gathered during identifying error with cause will allow us to know the reasons or causes for our failure to meet the expectations. This information will be used to initiate defect prevention activity by improving processes.

Error Classification: Type-wise
Defect may fall into any of the following type:
  • Missing Features
  • Violation of Standards
  • Documentation Mismatch
  • Logical Error
  • Omission
  • Cosmetic
  • User Interface
  • Connectivity
  • Miscellaneous

The information about error so arrived at is being used to identify practices that may require immediate attention. This will allow us to improving methodology or techniques being used in carrying-out a task.

Error Classification: Orthogonal Defect Classification (ODC)
In Orthogonal Defect Classification (ODC), we do defect categorization into classes much like characterizing a point in a Cartesian system of orthogonal axes by its (x, y, z) coordinates.

Example: (Defect Origin Phase, Defect Cause, Defect Type); (Project Phase, Defect Type, Defect Severity), etc

ODC based defect categorization collectively point to the part of the process which needs attention. Thus it provides the closed match to one of the identified values, thereby communicating the intent of the change. By using this we can identify areas requiring attention.

To effectively use ODC, a good Classification Scheme for the Organization that allows learning from experience and provides a means of communicating experiences between projects shall be arrived at. Such a classification will allow accuracy of resolution of defects. Effective ODC scheme has at least three requirements:

  • Restrict ODC based classification to smaller set, that enables clear human interpretation
  • Maintain consistency in classification scheme between the stages. So that it will be easy to look at trends across stages.
  • Make the classification independent of the specifics of a product or organization thereby maintains uniformity across products.
Khannur's Book
Arunkumar Khannur, Software Testing - Techniques and Applications, Published by Pearson Publications, 2011 (ISBN:978-81-317-5836-6; Pages:341 + xxii)
Follow Khannur
Khannur's Company
ISQT Process & Consulting Services Pvt. Ltd., Bangalore, INDIA
Khannur's Software Testing Forum
 Contact Khannur
ISQT Process & Consulting Services Pvt. Ltd.
#732, 1st Floor, 12th Main,
3rd Block, Rajajinagar,
Bangalore - 560010, INDIA
Phone: +91 80 23012511
Skype: arun.isqt