Pairwise Comparison-- The Analytic Hierarchy Process

The Analytic Hierarchy Process (AHP) is a method developed by T. L. Saaty for hierarchically decomposing complex judgments and, via simple comparisons among the components, deriving numerical scores representing their relative importance or value. One of the significant strengths of AHP is that it can measure the degree of inconsistency present in the pairwise judgments, and thereby help ensure that only justifiable rankings are used as the basis for audit plans.

Assume that three risk factors are identified as being appropriate for measuring the degree of risk/concern/exposure associated with audit units. All three risk factors may apply to each and every auditable unit within the organization. So, each audit unit must be evaluated with respect to each risk factor as indicated by the crisscrossing lines in Figure 5.

Figure 5: Analytic Hierarchy Structure of Risk Assessment

Example

With reference to Figure 5, assume that the objective is to minimize losses, as represented by risk to the firm as a whole. Further, assume that three audit units are being evaluated using three risk factors: Size, Quality of Internal Control and Complexity of Operations.

For each audit unit, this would result in the following three sets of pairwise comparisons:

With respect to each audit unit the rater(s) might be asked, "Which risk factor is more important? Risk factor 1 or risk factor 2?" "By how much?" For each audit unit, all pairs of risk factors are compared, one pair at a time, and a number from 1 to 9 is assigned to the one representing greater concern using a rating scale such as the one illustrated in Figure 7 and using a format such as the one illustrated in Figure 6. By making these simple pairwise judgments, it is possible to fill out a table of such comparisons. AHP uses a mathematical technique, eigenvector scaling, for translating these pairwise ratings into numerical scores representing the importance or riskiness of each individual audit unit.

One of the significant strengths of AHP is that it can measure the degree of inconsistency present in the pairwise judgments, and thereby help ensure that only justifiable rankings are used as the basis for audit plans. For example, assuming that Quality of Internal Control was the most important of the three risk factor categories, followed by Size and Complexity of Operations, a consistent set of pairwise comparisons would be as follows:

In contrast, an inconsistent set of pairwise comparisons would be as follows:

Taking the above example further, numerical scores are assigned to represent the degree to which one risk factor category is more important than another. A consistent set of ratings is:

In contrast, an inconsistent set of pairwise comparisons would be:

Figure 6: A Format for Recording Pairwise Comparisons

Figure 7: AHP Response Scale

Base Comparison

The is method is similar to the Pairwise Comparison method except that some factor is chosen to represent a base for comparison and all other factors are evaluated in comparison with this base.

Example

Assuming Size was selected to be the Base for Comparison, for each audit unit, this would result in the following three sets of pairwise comparisons:

Figure 8: A Format for Recording Base Comparison Ratings

These pairwise judgments are relatively simple to make; however, the base comparison approach lacks the built-in inconsistency checks of AHP which incorporates safeguards to ensure a reliable set of ratings. On the other hand, AHP's applicability to very large organizations may be limited because of the need for an excessive number of comparisons, whereas the other methods make a more modest demand on planning.

Group Judgments

Research has shown that groups can often make superior judgments than individuals. Groups can be nominal or interactive, face-to-face, or remote, used to working together or anonymous, and so on. The Analytic Hierarchy Process discussed in the previous section was designed for use by interacting groups.

Don't: choose too many factors, poor scales for rating the factors, inappropriate methods of eliciting factor ratings.

Do: choose factors that are applicable, and quantitative scales (e.g., scale of 0-100, 0-9, etc.) that are consistent across various sections of the audit universe. Otherwise, they will yield inconsistent and non-comparable scores.

Usually, subjective judgment about the relative importance of risk factors cannot be avoided, especially when the benefits from auditing are intangible or difficult to predict. Depending on organizational size and characteristics, a combination of methods can be applied.

Regardless of the methods used, it is important to predefine clear guidelines for evaluating risk factors properly. Validate ratings.

Group process should be used to the maximum extent possible; e.g., have a few senior auditors go through the process together or independently correlating their ratings, identifying areas of strong disagreement. Disagreements should be discussed and a consensus reached. Alternatively, collect judgments from individuals then combine them into an overall group assessment. Such mathematically combined groups (i.e., individual judgments mathematically combined into a group score) have been found to be often superior to individual judgments.

Wherever possible, auditee and managerial personnel should be involved in carrying out some or all of the risk assessment, since often they are in the best position to perceive problems as they develop, rather than after the fact. This can be in the form of workshops to identify risk factors, surveys used to capture risk judgments, and feedback sessions designed to evaluate past audit coverage plans. Such co-operation can enhance communication between auditors and auditees, enhance mutual respect, and benefit the entire planning process.

by

J. Efrim Boritz

School of Accountancy, University of Waterloo, Waterloo, Canada N2L 3G1

© Copyright 1992, 1993 All Rights Reserved