Sections of this topic
Evaluating Training and Results (ROI of Training)
© Copyright Carter
McNamara, MBA, PhD, Authenticity Consulting, LLC.
Sections of This Topic Include
Preparation for Evaluating Training Activities and Results
Perspective on Evaluating Training
Suggestions for Evaluating Training
One Approach to Calculate Return on Investment (ROI) of Training
Additional Resources to Guide Evaluation of Your Training
Also consider
Related Library Topics
Learn More in the Library’s Blogs Related to Evaluating Training and Result (ROI)
In addition to the articles on this current page, also see the following blogs
that have posts related to Evaluating Training and Results (ROI). Scan down
the blog’s page to see various posts. Also see the section “Recent Blog
Posts” in the sidebar of the blog or click on “next” near the
bottom of a post in the blog. The blog also links to numerous free related resources.
Library’s Career
Management Blog
Library’s Human Resources
Blog
Library’s Leadership Blog
Library’s Supervision
Blog
Library’s
Training and Development Blog
Preparation for Evaluating Training Activities
and Results
The last phase of the ADDIE model of instructional design, or systematic training,
is evaluation. However, the evaluation really should have started even during
the previous phase — the implementation phase — because the evaluation is
of both the activities of the trainer as they are being implemented and of the
results of the training as it nears an end or is finished. Evaluation includes
getting ongoing feedback, e.g., from the learner, trainer and learner’s supervisor,
to improve the quality of the training and identify if the learner achieved
the goals of the training.
- Before progressing through the guidelines in this topic, the reader would
benefit from first reviewing the information about formal and systematic training,
especially the ADDIE model, at Formal
Training Processes — Instructional Systems Design (ISD) and ADDIE. - Then scan the contents of the fourth phase of the ADDIE model systematic
planning of training, Implementing
Your Training Plan. (This evaluation phase is the fifth phase of the ADDIE
model.) - Also, note that there is a document, Complete
Guidelines to Design Your Training Plan, that condenses the guidelines
from the various topics about training plans to guide you to develop a training
plan. That document also provides a Framework
to Design Your Training Plan that you can use to document the various
aspects of your plan
Perspective on Evaluating
Training
Evaluation is often looked at from four different levels (the “Kirkpatrick
levels”) listed below. Note that the farther down the list, the more valid
the evaluation.
- Reaction – What does the learner feel about the training?
- Learning – What facts, knowledge, etc., did the learner gain?
- Behaviors – What skills did the learner develop, that is,
what new information is the learner using on the job? - Results or effectiveness – What results occurred, that is,
did the learner apply the new skills to the necessary tasks in
the organization and, if so, what results were achieved?
Although level 4, evaluating results and effectiveness, is
the most desired result from training, it’s usually the most difficult
to accomplish. Evaluating effectiveness often involves the use
of key performance measures — measures you can see, e.g., faster
and more reliable output from the machine after the operator has
been trained, higher ratings on employees’ job satisfaction questionnaires
from the trained supervisor, etc. This is where following sound
principles of performance management is of great benefit.
Stephanie Mooshegian on Making the Most Out of
Kirkpatrick’s Level 1 Measurement in the Classroom
Suggestions for Evaluating Training
Typically, evaluators look for validity, accuracy and reliability
in their evaluations. However, these goals may require more time,
people and money than the organization has. Evaluators are also
looking for evaluation approaches that are practical and relevant.
Training and development activities can be evaluated before,
during and after the activities. Consider the following very basic
suggestions:
Before the Implementation Phase
- Will the selected training and development methods really
result in the employee’s learning the knowledge and skills needed
to perform the task or carry out the role? Have other employee’s
used the methods and been successful? - Consider applying the methods to a highly skilled employee.
Ask the employee of their impressions of the methods. - Do the methods conform to the employee’s preferences and
learning styles? Have the employee briefly review the methods,
e.g., documentation, overheads, etc. Does the employee experience
any difficulties understanding the methods?
During Implementation of Training
- Ask the employee how they’re doing. Do they understand what’s
being said? - Periodically conduct a short test, e.g., have the employee
explain the main points of what was just described to him, e.g.,
in the lecture. - Is the employee enthusiastically taking part in the activities?
Is he or she coming late and leaving early. It’s surprising how
often learners will leave a course or workshop and immediately
complain that it was a complete waste of their time. Ask the
employee to rate the activities from 1 to 5, with 5 being the
highest rating. If the employee gives a rating of anything less
than 5, have the employee describe what could be done to get
a 5.
After Completion of the Training
- Give him or her a test before and after the training and
development, and compare the results? - Interview him or her before and after, and compare results?
- Watch him or her perform the task or conduct the role?
- Assign an expert evaluator from inside or outside the organization
to evaluate the learner’s knowledge and skills?
One Approach to Calculate Return On Investment
(ROI)
(This section was written by Leigh
Dudley. The section mentions HRD — activities of human resource development
— but the guidelines are as applicable to training and development.)
The calculation of ROI in [training and development] or HRD begins with the
basic model, where sequential steps simplify a potentially complicated process.
The ROI process model provides a systematic approach to ROI calculations.
The step-by-step approach keeps the process manageable so that users can tackle
one issue at a time. The model also emphasizes that this is a logical process
that flows from one step to another. ROI calculation to another provides consistency,
understanding, and credibility. Each step of the model is briefly described
below.
Collecting Post-Program Data
Data collection is central to the ROI process and is the starting point of
the ROI process . Although the ROI analysis is (or should be) planned early
in the training and development cycle, the actual ROI calculation begins with
data collection. (Additional information on planning for the ROI analysis is
presented later under “Essential Planning Steps).
The HRD staff should collect both hard data (representing output, quality,
cost, and time) and soft data (including work habits, work climate, and attitudes).
Collect Level 4 data using a variety of the methods as follows:
- Follow-up Questionnaires – Administer follow-up
questionnaires to uncover specific applications of training. Participants
provide responses to a variety of types of open-ended and forced response
questions. - Use questionnaires to capture both Level 3 and Level 4 data. The example
below shows a series of level 4 impact questions contained in a follow-up
questionnaire for evaluating an automotive manufacturer’s sales training
program in Europe, with appropriate responses. HRD practitioners can use the
data in an ROI analysis - Program Assignments – Program assignments are useful
for simple, short-term projects. Participants complete the assignment on the
job, using the skills or knowledge learned in the program. Report completed
assignments as evaluation information, which often contains Level 3/Level
4 data. Convert Level 4 data to monetary values and compare the data to cost
to develop the ROI - Action Plans – Developed in training and development
programs, action plans on the job should be implemented after the program
is completed. A follow-up of the plans provides evaluation information. Level
3/Level 4 data are collected with action plans, and the HRD staff can develop
the ROI from the Level 4 data. - Performance Contracts – Developed prior to conducting
the program and when the participant, the participant’s supervisor,
and the instructor all agree on planned specific out-comes from the training,
performance contracts outline how the program will be implemented. Performance
contracts usually collect both Level 3/and Level 4 data and are designed and
analyzed in the same way as action plans. - Performance Monitoring – As the most beneficial method
to collect Level 4 data, performance monitoring is useful when HRD personnel
examine various business performance records and operational data for improvement.
The important challenge in this step is to select the data collection method
or methods that are appropriate for both the setting and the specific program
and the time and budget constraints.
Isolating the Effects of Training
Isolating the effects of training is an often overlooked issue in evaluations.
In this step of the ROI process, explore specific techniques to determine the
amount of output performance directly related to the program. This step is essential
because many factors influence performance data after training. The specific
techniques of this step will pinpoint the amount of improvement directly related
to the program, increasing the accuracy and credibility of the ROI calculation.
Collectively, the following techniques provide a comprehensive set of tools
to tackle the important and critical issue of isolating the effects of training.
- Control Group – use a control group arrangement
to isolate training impact. With this technique, one group receives training
while another similar, group does not receive training. The difference in
the performance of the two groups is attributed to the training program. When
properly set up and implemented, control group arrangement is the most effective
way to isolate the effects of training. - Impact Estimates – When the previous approach is
not feasible, estimating the impact of training on the output variables is
another approach and can be accomplished on the following 4 levels. - Participants – estimate the amount of improvement
related to training. In this approach, provide participants with the total
amount of improvement, on a pre- and post-program basis, and ask them to indicate
the percent of the improvement that is actually related to the training program. - Supervisors – of participants estimate the impact
of training on the output variables. Present supervisors with the total amount
of improvement, and ask them to indicate the percent related to training. - Senior Managers – estimate the impact of training
by providing an estimate or adjustment to reflect the portion of the improvement
related to the training program. While perhaps inaccurate, having senior management
involved in this process develops ownership of the value and buy-in process. - Experts –estimate the impact of training on the performance
variable. Because these estimates are based on previous experience, experts
must be familiar with the type of training and the specific situation.
Customers sometimes provide input on the extent to which training has influenced
their decision to use a product or service. Although this approach has limited
applications, it can be quite useful in customer service and sales training.
Converting Data to Monetary Values
A number of techniques are available to convert data to monetary values; the
selection depends on the type of data and the situation.
- Convert output data to profit contribution or cost savings. With this technique,
output increases are converted to monetary value based on their unit contribution
to profit or the unit of cost reduction. These values are readily available
in most organizations and are seen as generally accepted standard values. - Calculate the cost of quality, and covert quality improvements directly
to cost savings. This standard value is available in many organizations for
the most common quality measures (such as rejects, rework, and scrap). - Use the participants’ wages and employee benefits as the value for
time in programs where employee time is saved. Because a variety of programs
focus on improving the time required to complete projects, processes, or daily
activities, the value of time becomes an important and necessary issue. The
use of total compensation per hour provides a conservative estimate for the
value of time. - Use historical costs when they are available for a specific variable. In
this case, use organizational cost data to establish the specific value of
an improvement. - Use internal and external experts, when available, to estimate a value for
an improvement. In this situation, the credibility of the estimate hinges
on the expertise and reputation of the individual. - Use external databases, when available, to estimate the value or cost of
data items. Research, government, and industry databases can provide important
for these values. The difficulty lies in finding a specific database related
to the situation. - Ask participants to estimate the value of the data item. For this approach
to be effective, participants must understand the process and be capable of
providing a value for the improvement. - Require supervisors and managers to provide estimates when they are willing
and capable of assigning values to the improvement. This approach is especially
useful when participants are not fully capable of providing this input or
in situations where supervisors or managers need to confirm or adjust the
participant’s estimate.
Converting data to monetary value is very important in the ROI model and is
absolutely necessary to determine the monetary benefits from a training program.
The process is challenging, particularly with the conversion of soft data, but
can be methodically accomplished using one or more of the above techniques.
Tabulating Program Costs
The other part of the equation in a cost/benefit analysis is the cost of the
program. Tabulating the costs involves monitoring or developing all of the related
costs of the program targeted for the ROI calculation. Include the following
items among the cost components.
- Cost to design and develop the program, possibly prorated over the expected
life of the program - Cost of all program materials provided to each participant
- Cost for the instructor/facilitator, including preparation time as well
as delivery time. - Cost of the facilities for the training program.
- Cost of travel, lodging and meals for the participants, if applicable.
- Salaries, plus employee benefits of the training function, allocated in
some convenient way.
In addition, specific cost related to the needs assessment and evaluation should
be included, if appropriate. The conservative approach is to include all of
these costs so that the total is fully loaded.
Calculating the ROI
Calculate the ROI using the program benefits and costs. The BCR is the program
benefits divided by costs:
- BCR = program benefits / program costs
- (Sometimes this ratio is stated as a cost/benefit ratio, although the formula
is the same as BCR).
The net benefits are the program benefits minus the costs:
- Net benefits = program benefits – program costs
The ROI uses the net benefits divided by programs costs:
- ROI (%) = net benefits / program costs x 100
Use the same basic formula in evaluating other investments where the ROI is
traditionally reported as earnings divided by investment. The ROI from some
training programs is high. For example, in sales training, supervisory training,
and managerial training, the ROI can be quite large, frequently over 100 percent,
while ROI value for technical and operator training may be lower.
Additional Resources to Guide Evaluation of Your Training
Evaluating
Training: There is no “Cookbook” Approach
Evaluating Training and Technical Assistance
Finding
the Bottom-Line Payoff for Training
Stakeholder
Approach to Evaluating Training
Calculating
ROI
Ensuring Behavior Change Occurs From Your Training
Program
Instruments
to Measure Post Training Effectiveness
ROA Formula / Return on Assets Calculation
Determining Training ROI
Survey
on Employer-Provided Formal Training Programs
Mary
Talbut on Testing What You Teach
Shonta Smith
on Student Ratings of Instruction: Evaluating the Professor
Camesha
Hill-Carter on Looking at Assessment in a Different Light
Training
Assessments: Personality Counts
Donald L Kirkpatrick’s training evaluation model
– the four levels of learning evaluation
Training
Brainstorm: Evaluating Trainers
Is Lecture Learning?
Seven Steps To Guarantee Great Training Results
Four Reasons for a Needs Assessment Survey Plus Two
5 Ways to Assess Training Results
Addressing the Dreams–Making Training More Effective
On Evaluating Standardized Tests
Evaluating Online Learning
Evaluating Online Learning
What Are the Signs of a High-Quality Online Course?
Learn More in the Library’s Blogs Related to This
Topic
In addition to the articles on this current page, also see the following blogs
that have posts related to this topic. Scan down the blog’s page to see various
posts. Also see the section “Recent Blog Posts” in the sidebar of
the blog or click on “next” near the bottom of a post in the blog.
The blog also links to numerous free related resources.
Library’s Career
Management Blog
Library’s Human Resources
Blog
Library’s Leadership Blog
Library’s Supervision
Blog
Library’s
Training and Development Blog
Go to main Training
and Development page.
For the Category of Training and Development:
To round out your knowledge of this Library topic, you may
want to review some related topics, available from the link below.
Each of the related topics includes free, online resources.
Also, scan the Recommended Books listed below. They have been
selected for their relevance and highly practical nature.
