Pay-for-Performance

Reducing Errors

Several options are available to reduce errors in rating scales, including:

• factor choice
• behavioral checklists
• critical incident method

Factor choice

A focus on behavior and/or results improves the actual observation ability of the raters; they can focus on what they have observed.

Systems that include the raters in their development have a better chance of success.

Behavioral checklists

Raters need to record, NOT evaluate. You can encourage this by having raters use behavioral checklists. Behavioral checklists are a set of statements about the behaviors that an employee might engage in on the job.

The rater's task is to indicate whether the employee does or does not engage in the behavior. This puts the rater in the position of recording behavior more than evaluating it, and should lead to more reliable reporting.

A behavioral checklist can be made more complex by requiring the rater to indicate how much the employee engages in the behavior.

Forced choice

Forced choice is a special form of a behavioral checklist. In the forced choice method, the rater is presented with a set of behaviors and is requested to choose the most descriptive and least descriptive behaviors of the employee. In a set of 4 items, 2 appear favorable and 2 unfavorable. However, only 1 of the favorable items adds to the total score and only 1 of the negative ones detracts from it. The value of the items is determined statistically by an item analysis of successful and unsuccessful employees. The scores are not known to the rater, who is in essence rating blind. This last feature is intended to reduce the constant errors of rating scales.

Origin. The armed forces developed the forced choice method where the problem of leniency had led to everyone being rated excellent.

Advantages. The forced choice method reduces error and has the advantage of having the rater record rather than evaluate.

Disadvantages. The secrecy feature of forced choice leaves raters questioning how they rated the employee. This is uncomfortable and leads to resistance to and subversion of the method. As a method of providing feedback to the employee, forced choice is not useful, since neither rater nor the employee being rated knows what the important behaviors are. In fact, the armed forces have abandoned its use.

In the end, any behavioral checklist that evaluates employee behaviors that provide an overall score would be useful in establishing the relative performance of employees.