Developed by Authenticity Consulting, LLC
Evaluating Training and Results (ROI of Training)
© Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC.
Sections of This Topic Include
Preparation for Evaluating Training Activities and Results
Perspective on Evaluating Training
Suggestions for Evaluating Training
One Approach to Calculate Return on Investment (ROI) of Training
Additional Resources to Guide Evaluation of Your Training
Also see
Related Library Topics
Also See the Library's Blogs Related to Evaluating Training and Result (ROI)
In addition to the articles on this current page, also see the following blogs that have posts related to Evaluating Training and Results (ROI). Scan down the blog's page to see various posts. Also see the section "Recent Blog Posts" in the sidebar of the blog or click on "next" near the bottom of a post in the blog. The blog also links to numerous free related resources.
Library's Career
Management Blog
Library's Human Resources
Blog
Library's Leadership Blog
Library's Supervision
Blog
Library's
Training and Development Blog
Preparation for Evaluating Training Activities and Results
The last phase of the ADDIE model of instructional design, or systematic training, is evaluation. However, the evaluation really should have started even during the previous phase -- the implementation phase -- because the evaluation is of both the activities of the trainer as they are being implemented and of the results of the training as it nears an end or is finished. Evaluation includes getting ongoing feedback, e.g., from the learner, trainer and learner's supervisor, to improve the quality of the training and identify if the learner achieved the goals of the training.
- Before progressing through the guidelines in this topic, the reader would benefit from first reviewing the information about formal and systematic training, especially the ADDIE model, at Formal Training Processes -- Instructional Systems Design (ISD) and ADDIE.
- Then scan the contents of the fourth phase of the ADDIE model systematic planning of training, Implementing Your Training Plan. (This evaluation phase is the fifth phase of the ADDIE model.)
- Also, note that there is a document, Complete Guidelines to Design Your Training Plan, that condenses the guidelines from the various topics about training plans to guide you to develop a training plan. That document also provides a Framework to Design Your Training Plan that you can use to document the various aspects of your plan
Perspective on Evaluating Training
Evaluation is often looked at from four different levels (the "Kirkpatrick levels") listed below. Note that the farther down the list, the more valid the evaluation.
- Reaction - What does the learner feel about the training?
- Learning - What facts, knowledge, etc., did the learner gain?
- Behaviors - What skills did the learner develop, that is, what new information is the learner using on the job?
- Results or effectiveness - What results occurred, that is, did the learner apply the new skills to the necessary tasks in the organization and, if so, what results were achieved?
Although level 4, evaluating results and effectiveness, is the most desired result from training, it's usually the most difficult to accomplish. Evaluating effectiveness often involves the use of key performance measures -- measures you can see, e.g., faster and more reliable output from the machine after the operator has been trained, higher ratings on employees' job satisfaction questionnaires from the trained supervisor, etc. This is where following sound principles of performance management is of great benefit.
Stephanie Mooshegian on Making the Most Out of
Kirkpatrick's Level 1 Measurement in the Classroom
Suggestions for Evaluating Training
Typically, evaluators look for validity, accuracy and reliability in their evaluations. However, these goals may require more time, people and money than the organization has. Evaluators are also looking for evaluation approaches that are practical and relevant.
Training and development activities can be evaluated before, during and after the activities. Consider the following very basic suggestions:
Before the Implementation Phase
- Will the selected training and development methods really result in the employee's learning the knowledge and skills needed to perform the task or carry out the role? Have other employee's used the methods and been successful?
- Consider applying the methods to a highly skilled employee. Ask the employee of their impressions of the methods.
- Do the methods conform to the employee's preferences and learning styles? Have the employee briefly review the methods, e.g., documentation, overheads, etc. Does the employee experience any difficulties understanding the methods?
During Implementation of Training
- Ask the employee how they're doing. Do they understand what's being said?
- Periodically conduct a short test, e.g., have the employee explain the main points of what was just described to him, e.g., in the lecture.
- Is the employee enthusiastically taking part in the activities? Is he or she coming late and leaving early. It's surprising how often learners will leave a course or workshop and immediately complain that it was a complete waste of their time. Ask the employee to rate the activities from 1 to 5, with 5 being the highest rating. If the employee gives a rating of anything less than 5, have the employee describe what could be done to get a 5.
After Completion of the Training
- Give him or her a test before and after the training and development, and compare the results?
- Interview him or her before and after, and compare results?
- Watch him or her perform the task or conduct the role?
- Assign an expert evaluator from inside or outside the organization to evaluate the learner's knowledge and skills?
One Approach to Calculate Return On Investment (ROI)
(This section was written by Leigh Dudley. The section mentions HRD -- activities of human resource development -- but the guidelines are as applicable to training and development.)
The calculation of ROI in [training and development] or HRD begins with the basic model, where sequential steps simplify a potentially complicated process. The ROI process model provides a systematic approach to ROI calculations.
The step-by-step approach keeps the process manageable so that users can tackle one issue at a time. The model also emphasizes that this is a logical process that flows from one step to another. ROI calculation to another provides consistency, understanding, and credibility. Each step of the model is briefly described below.
Collecting Post-Program Data
Data collection is central to the ROI process and is the starting point of the ROI process . Although the ROI analysis is (or should be) planned early in the training and development cycle, the actual ROI calculation begins with data collection. (Additional information on planning for the ROI analysis is presented later under “Essential Planning Steps).
The HRD staff should collect both hard data (representing output, quality, cost, and time) and soft data (including work habits, work climate, and attitudes). Collect Level 4 data using a variety of the methods as follows:
- Follow-up Questionnaires – Administer follow-up questionnaires to uncover specific applications of training. Participants provide responses to a variety of types of open-ended and forced response questions.
- Use questionnaires to capture both Level 3 and Level 4 data. The example below shows a series of level 4 impact questions contained in a follow-up questionnaire for evaluating an automotive manufacturer’s sales training program in Europe, with appropriate responses. HRD practitioners can use the data in an ROI analysis
- Program Assignments – Program assignments are useful for simple, short-term projects. Participants complete the assignment on the job, using the skills or knowledge learned in the program. Report completed assignments as evaluation information, which often contains Level 3/Level 4 data. Convert Level 4 data to monetary values and compare the data to cost to develop the ROI
- Action Plans – Developed in training and development programs, action plans on the job should be implemented after the program is completed. A follow-up of the plans provides evaluation information. Level 3/Level 4 data are collected with action plans, and the HRD staff can develop the ROI from the Level 4 data.
- Performance Contracts – Developed prior to conducting the program and when the participant, the participant’s supervisor, and the instructor all agree on planned specific out-comes from the training, performance contracts outline how the program will be implemented. Performance contracts usually collect both Level 3/and Level 4 data and are designed and analyzed in the same way as action plans.
- Performance Monitoring – As the most beneficial method to collect Level 4 data, performance monitoring is useful when HRD personnel examine various business performance records and operational data for improvement.
The important challenge in this step is to select the data collection method or methods that are appropriate for both the setting and the specific program and the time and budget constraints.
Isolating the Effects of Training
Isolating the effects of training is an often overlooked issue in evaluations. In this step of the ROI process, explore specific techniques to determine the amount of output performance directly related to the program. This step is essential because many factors influence performance data after training. The specific techniques of this step will pinpoint the amount of improvement directly related to the program, increasing the accuracy and credibility of the ROI calculation. Collectively, the following techniques provide a comprehensive set of tools to tackle the important and critical issue of isolating the effects of training.
- Control Group – use a control group arrangement to isolate training impact. With this technique, one group receives training while another similar, group does not receive training. The difference in the performance of the two groups is attributed to the training program. When properly set up and implemented, control group arrangement is the most effective way to isolate the effects of training.
- Impact Estimates – When the previous approach is not feasible, estimating the impact of training on the output variables is another approach and can be accomplished on the following 4 levels.
- Participants – estimate the amount of improvement related to training. In this approach, provide participants with the total amount of improvement, on a pre- and post-program basis, and ask them to indicate the percent of the improvement that is actually related to the training program.
- Supervisors – of participants estimate the impact of training on the output variables. Present supervisors with the total amount of improvement, and ask them to indicate the percent related to training.
- Senior Managers – estimate the impact of training by providing an estimate or adjustment to reflect the portion of the improvement related to the training program. While perhaps inaccurate, having senior management involved in this process develops ownership of the value and buy-in process.
- Experts –estimate the impact of training on the performance variable. Because these estimates are based on previous experience, experts must be familiar with the type of training and the specific situation.
Customers sometimes provide input on the extent to which training has influenced their decision to use a product or service. Although this approach has limited applications, it can be quite useful in customer service and sales training.
Converting Data to Monetary Values
A number of techniques are available to convert data to monetary values; the selection depends on the type of data and the situation.
- Convert output data to profit contribution or cost savings. With this technique, output increases are converted to monetary value based on their unit contribution to profit or the unit of cost reduction. These values are readily available in most organizations and are seen as generally accepted standard values.
- Calculate the cost of quality, and covert quality improvements directly to cost savings. This standard value is available in many organizations for the most common quality measures (such as rejects, rework, and scrap).
- Use the participants’ wages and employee benefits as the value for time in programs where employee time is saved. Because a variety of programs focus on improving the time required to complete projects, processes, or daily activities, the value of time becomes an important and necessary issue. The use of total compensation per hour provides a conservative estimate for the value of time.
- Use historical costs when they are available for a specific variable. In this case, use organizational cost data to establish the specific value of an improvement.
- Use internal and external experts, when available, to estimate a value for an improvement. In this situation, the credibility of the estimate hinges on the expertise and reputation of the individual.
- Use external databases, when available, to estimate the value or cost of data items. Research, government, and industry databases can provide important for these values. The difficulty lies in finding a specific database related to the situation.
- Ask participants to estimate the value of the data item. For this approach to be effective, participants must understand the process and be capable of providing a value for the improvement.
- Require supervisors and managers to provide estimates when they are willing and capable of assigning values to the improvement. This approach is especially useful when participants are not fully capable of providing this input or in situations where supervisors or managers need to confirm or adjust the participant’s estimate.
Converting data to monetary value is very important in the ROI model and is absolutely necessary to determine the monetary benefits from a training program. The process is challenging, particularly with the conversion of soft data, but can be methodically accomplished using one or more of the above techniques.
Tabulating Program Costs
The other part of the equation in a cost/benefit analysis is the cost of the program. Tabulating the costs involves monitoring or developing all of the related costs of the program targeted for the ROI calculation. Include the following items among the cost components.
- Cost to design and develop the program, possibly prorated over the expected life of the program
- Cost of all program materials provided to each participant
- Cost for the instructor/facilitator, including preparation time as well as delivery time.
- Cost of the facilities for the training program.
- Cost of travel, lodging and meals for the participants, if applicable.
- Salaries, plus employee benefits of the training function, allocated in some convenient way.
In addition, specific cost related to the needs assessment and evaluation should be included, if appropriate. The conservative approach is to include all of these costs so that the total is fully loaded.
Calculating the ROI
Calculate the ROI using the program benefits and costs. The BCR is the program benefits divided by costs:
- BCR = program benefits / program costs
- (Sometimes this ratio is stated as a cost/benefit ratio, although the formula is the same as BCR).
The net benefits are the program benefits minus the costs:
- Net benefits = program benefits – program costs
The ROI uses the net benefits divided by programs costs:
- ROI (%) = net benefits / program costs x 100
Use the same basic formula in evaluating other investments where the ROI is traditionally reported as earnings divided by investment. The ROI from some training programs is high. For example, in sales training, supervisory training, and managerial training, the ROI can be quite large, frequently over 100 percent, while ROI value for technical and operator training may be lower.
Additional Resources to Guide Evaluation of Your Training
The
Evaluation of Training
Evaluating
Training: There is no "Cookbook" Approach
Evaluating Training and Technical Assistance
Finding
the Bottom-Line Payoff for Training
Stakeholder
Approach to Evaluating Training
Evaluating training: White paper and recommendations
Calculating
ROI
Ensuring Behavior Change Occurs From Your Training
Program
Instruments
to Measure Post Training Effectiveness
Assessing the Return on Asset (ROI) of Training
(comprehensive overview of costs and benefits)
Determining Training ROI
Survey
on Employer-Provided Formal Training Programs
Mary
Talbut on Testing What You Teach
Shonta Smith
on Student Ratings of Instruction: Evaluating the Professor
Camesha
Hill-Carter on Looking at Assessment in a Different Light
Training
Assessments: Personality Counts
Donald L Kirkpatrick's training evaluation model
- the four levels of learning evaluation
Training
Brainstorm: Evaluating Trainers
Is Lecture Learning?
Seven Steps To Guarantee Great Training Results
Four Reasons for a Needs Assessment Survey Plus Two
5 Ways to Assess Training Results
Addressing the Dreams–Making Training More Effective
On Evaluating Standardized Tests
Evaluating Online Learning
Evaluating Online Learning
What does a high quality online course look like?
Also See the Library's Blogs Related to this Topic
In addition to the articles on this current page, also see the following blogs that have posts related to this topic. Scan down the blog's page to see various posts. Also see the section "Recent Blog Posts" in the sidebar of the blog or click on "next" near the bottom of a post in the blog. The blog also links to numerous free related resources.
Library's Career
Management Blog
Library's Human Resources
Blog
Library's Leadership Blog
Library's Supervision
Blog
Library's
Training and Development Blog
Go to main Training and Development page.
For the Category of Training and Development:
To round out your knowledge of this Library topic, you may
want to review some related topics, available from the link below.
Each of the related topics includes free, online resources.
Also, scan the Recommended Books listed below. They have been
selected for their relevance and highly practical nature.
Related Library Topics
Recommended Books - Training and Development
Basics and General Information
Orienting and Training Employees
Basics and General Information
-
Field Guide to Leadership and Supervision in Business
- by Carter McNamara, published by Authenticity Consulting, LLC. Provides step-by-step, highly practical guidelines to recruit, utilize and evaluate the best employees for your business. Includes guidelines to effectively lead yourself (as Board member or employee), other individuals, groups and organizations. Includes guidelines to avoid burnout -- a very common problem among employees of small businesses. Many materials in this Library's topic about staffing are adapted from this book.
-
Field Guide to Leadership and Supervision With Nonprofit Staff
- by Carter McNamara, published by Authenticity Consulting, LLC. Provides step-by-step, highly practical guidelines to recruit, utilize and evaluate the best staff members for your nonprofit. Includes guidelines to effectively lead yourself (as Board member or staff member), other individuals, groups and organizations. Includes guidelines to avoid burnout -- a very common problem among nonprofit staff. Many materials in this Library's topic about staffing are adapted from this book.
The following books are recommended because of their highly practical nature and often because they include a wide range of information about this Library topic. To get more information about each book, just click on the image of the book. Also, a convenient search window is included below the rows of boxes.
Orienting and Training Employees
The following books are recommended because of their highly practical nature and often because they include a wide range of information about this Library topic. To get more information about each book, just click on the image of the book. Also, a convenient search window is included below the rows of boxes.
