Phase V - Evaluation

Learning Objectives:

1. Be able to evaluate the effectiveness of a training program using the four levels of measurement described by Kirkpatrick.
2. Be able to describe the six basic data collection methods and the most appropriate level of analysis for each level of evaluation.
3. Be able to discuss the various research designs and the most appropriate design for your particular training program.

 Evaluating Training Programs

 This discussion focuses on Kirkpatrick’s four levels (reaction, learning, behavior, and results) and is taken largely from Chapter 5 “Evaluating Training Programs” from Wexley and Latham (2002).

 Reactions – Reaction criteria measure how well the participants liked the program, including its content, the trainer, the methods used, and the surroundings in which the training took place.  This is often measured with a questionnaire. 

1. Some reaction measures correlate moderately with immediate learning.
2. Positive reactions help ensure organizational support.
3. The measures can help plan future programs.
4. Favorable reactions enhance a trainee’s motivation to learn
5. It is helpful to compare different groups of trainees
6. Reactions measures should be taken again several months after the training for a more realistic assessment of the impact on the trainee’s job.

 Learning – Learning criteria assesses the knowledge gained by the trainees. The trainees’ learning correlates moderately with immediate job behavior. Typically knowledge is measured with paper and pencil tests. Performance tests are used to determine if trainees have mastered a particular skill.   It is important to remember that training can focus on three types of learning outcomes: knowledge (declarative, procedural and strategic), skills (compilation and automaticity), and attitudes.  Depending on how long the training is, it might be worth assessing learning periodically throughout the training. If training is not progressing as expected, some modification may be in order.

 Behavior on the Job – Once it has been determined that learning has taken place, the next step is to determine whether the training has transferred to the job. The primary sources of data are interviews, questionnaires, direct observations, and archival records of performance. 

 Results Criteria – The objectives of training are developed to solve an organizational problem.  It is important to assess this final level because it is the reason for doing the training in the first place.  The links between organizational results, job behavior, and trainee KSAs should have been articulated in the training needs analysis.  An important part of this phase is the return on investment.

Evaluation Chart 1 - Data Collection Methods

Training Evaluation Designs

A wide variety of text provide excellent information on evaluation designs. However in many situations many designs are impractical in typical organizational settings. As an example, finding the time or resources to create a control group is difficult. As a result we find that the most common designs are the posttest only and the pretest/posttest methods. While these are imperfect designs they still offer some rigor to the evaluation process.

• Posttest Only. This occurs when training is provided and then a posttest is given. If the evaluation has two goals, to determine if a change has taken place and to determine if a level of competency has been achieved this method may be acceptable. If an adequate needs analysis has been conducted you may already have pre-test information, so the postest method may be all that is needed.
• Pretest/Posttest. This occurs when the learners are tested prior to attending training and then tested again after completion of training. While is normally considered a better design than the pretest only, there are criticisms. Without a control group, it is difficult to determine if the training had an impact on the outcomes or other factors were involved.

Most training texts  research methods texts cover the other more complex designs in detail and will not be explained in this discussion. They are the Pottest Olyl with Control Group, Pretest/Posttest with Control Group, the Time Series Design, the Solomon 4 Group Design.

Resources used for this discussion:

Wexley, Kenneth N. and Gary P. Latham, Developing and Training Human Resources in Organizations, 3rd Ed. Prentice hall, Upper Saddle River, NJ 07458 (2002)

Return to Index