Evaluating Training and Results (ROI of Training)

Sections of this topic

    Evaluating Training and Results (ROI of Training)

    © Copyright Carter
    McNamara, MBA, PhD, Authenticity Consulting, LLC

    Sections of This Topic Include

    Preparation for Evaluating Training Activities and Results
    Perspective on Evaluating Training
    Suggestions for Evaluating Training
    One Approach to Calculate Return on Investment (ROI) of Training
    Additional Resources to Guide Evaluation of Your Training

    Also consider
    Related Library Topics

    Learn More in the Library’s Blogs Related to Evaluating Training and Result (ROI)

    In addition to the articles on this current page, also see the following blogs
    that have posts related to Evaluating Training and Results (ROI). Scan down
    the blog’s page to see various posts. Also see the section “Recent Blog
    Posts” in the sidebar of the blog or click on “next” near the
    bottom of a post in the blog. The blog also links to numerous free related resources.

    Library’s Career
    Management Blog

    Library’s Human Resources

    Library’s Leadership Blog
    Library’s Supervision

    Training and Development Blog

    Preparation for Evaluating Training Activities
    and Results

    The last phase of the ADDIE model of instructional design, or systematic training,
    is evaluation. However, the evaluation really should have started even during
    the previous phase — the implementation phase — because the evaluation is
    of both the activities of the trainer as they are being implemented and of the
    results of the training as it nears an end or is finished. Evaluation includes
    getting ongoing feedback, e.g., from the learner, trainer and learner’s supervisor,
    to improve the quality of the training and identify if the learner achieved
    the goals of the training.

    Perspective on Evaluating

    Evaluation is often looked at from four different levels (the “Kirkpatrick
    levels”) listed below. Note that the farther down the list, the more valid
    the evaluation.

    1. Reaction – What does the learner feel about the training?
    2. Learning – What facts, knowledge, etc., did the learner gain?
    3. Behaviors – What skills did the learner develop, that is,
      what new information is the learner using on the job?
    4. Results or effectiveness – What results occurred, that is,
      did the learner apply the new skills to the necessary tasks in
      the organization and, if so, what results were achieved?

    Although level 4, evaluating results and effectiveness, is
    the most desired result from training, it’s usually the most difficult
    to accomplish. Evaluating effectiveness often involves the use
    of key performance measures — measures you can see, e.g., faster
    and more reliable output from the machine after the operator has
    been trained, higher ratings on employees’ job satisfaction questionnaires
    from the trained supervisor, etc. This is where following sound
    principles of performance management is of great benefit.

    Stephanie Mooshegian on Making the Most Out of
    Kirkpatrick’s Level 1 Measurement in the Classroom

    Suggestions for Evaluating Training

    Typically, evaluators look for validity, accuracy and reliability
    in their evaluations. However, these goals may require more time,
    people and money than the organization has. Evaluators are also
    looking for evaluation approaches that are practical and relevant.

    Training and development activities can be evaluated before,
    during and after the activities. Consider the following very basic

    Before the Implementation Phase

    • Will the selected training and development methods really
      result in the employee’s learning the knowledge and skills needed
      to perform the task or carry out the role? Have other employee’s
      used the methods and been successful?
    • Consider applying the methods to a highly skilled employee.
      Ask the employee of their impressions of the methods.
    • Do the methods conform to the employee’s preferences and
      learning styles? Have the employee briefly review the methods,
      e.g., documentation, overheads, etc. Does the employee experience
      any difficulties understanding the methods?

    During Implementation of Training

    • Ask the employee how they’re doing. Do they understand what’s
      being said?
    • Periodically conduct a short test, e.g., have the employee
      explain the main points of what was just described to him, e.g.,
      in the lecture.
    • Is the employee enthusiastically taking part in the activities?
      Is he or she coming late and leaving early. It’s surprising how
      often learners will leave a course or workshop and immediately
      complain that it was a complete waste of their time. Ask the
      employee to rate the activities from 1 to 5, with 5 being the
      highest rating. If the employee gives a rating of anything less
      than 5, have the employee describe what could be done to get
      a 5.

    After Completion of the Training

    • Give him or her a test before and after the training and
      development, and compare the results?
    • Interview him or her before and after, and compare results?
    • Watch him or her perform the task or conduct the role?
    • Assign an expert evaluator from inside or outside the organization
      to evaluate the learner’s knowledge and skills?

    One Approach to Calculate Return On Investment

    (This section was written by Leigh
    . The section mentions HRD — activities of human resource development
    — but the guidelines are as applicable to training and development.)

    The calculation of ROI in [training and development] or HRD begins with the
    basic model, where sequential steps simplify a potentially complicated process.
    The ROI process model provides a systematic approach to ROI calculations.

    The step-by-step approach keeps the process manageable so that users can tackle
    one issue at a time. The model also emphasizes that this is a logical process
    that flows from one step to another. ROI calculation to another provides consistency,
    understanding, and credibility. Each step of the model is briefly described

    Collecting Post-Program Data

    Data collection is central to the ROI process and is the starting point of
    the ROI process . Although the ROI analysis is (or should be) planned early
    in the training and development cycle, the actual ROI calculation begins with
    data collection. (Additional information on planning for the ROI analysis is
    presented later under “Essential Planning Steps).

    The HRD staff should collect both hard data (representing output, quality,
    cost, and time) and soft data (including work habits, work climate, and attitudes).
    Collect Level 4 data using a variety of the methods as follows:

    • Follow-up Questionnaires – Administer follow-up
      questionnaires to uncover specific applications of training. Participants
      provide responses to a variety of types of open-ended and forced response
    • Use questionnaires to capture both Level 3 and Level 4 data. The example
      below shows a series of level 4 impact questions contained in a follow-up
      questionnaire for evaluating an automotive manufacturer’s sales training
      program in Europe, with appropriate responses. HRD practitioners can use the
      data in an ROI analysis
    • Program Assignments – Program assignments are useful
      for simple, short-term projects. Participants complete the assignment on the
      job, using the skills or knowledge learned in the program. Report completed
      assignments as evaluation information, which often contains Level 3/Level
      4 data. Convert Level 4 data to monetary values and compare the data to cost
      to develop the ROI
    • Action Plans – Developed in training and development
      programs, action plans on the job should be implemented after the program
      is completed. A follow-up of the plans provides evaluation information. Level
      3/Level 4 data are collected with action plans, and the HRD staff can develop
      the ROI from the Level 4 data.
    • Performance Contracts – Developed prior to conducting
      the program and when the participant, the participant’s supervisor,
      and the instructor all agree on planned specific out-comes from the training,
      performance contracts outline how the program will be implemented. Performance
      contracts usually collect both Level 3/and Level 4 data and are designed and
      analyzed in the same way as action plans.
    • Performance Monitoring – As the most beneficial method
      to collect Level 4 data, performance monitoring is useful when HRD personnel
      examine various business performance records and operational data for improvement.

    The important challenge in this step is to select the data collection method
    or methods that are appropriate for both the setting and the specific program
    and the time and budget constraints.

    Isolating the Effects of Training

    Isolating the effects of training is an often overlooked issue in evaluations.
    In this step of the ROI process, explore specific techniques to determine the
    amount of output performance directly related to the program. This step is essential
    because many factors influence performance data after training. The specific
    techniques of this step will pinpoint the amount of improvement directly related
    to the program, increasing the accuracy and credibility of the ROI calculation.
    Collectively, the following techniques provide a comprehensive set of tools
    to tackle the important and critical issue of isolating the effects of training.

    • Control Group – use a control group arrangement
      to isolate training impact. With this technique, one group receives training
      while another similar, group does not receive training. The difference in
      the performance of the two groups is attributed to the training program. When
      properly set up and implemented, control group arrangement is the most effective
      way to isolate the effects of training.
    • Impact Estimates – When the previous approach is
      not feasible, estimating the impact of training on the output variables is
      another approach and can be accomplished on the following 4 levels.
    • Participants – estimate the amount of improvement
      related to training. In this approach, provide participants with the total
      amount of improvement, on a pre- and post-program basis, and ask them to indicate
      the percent of the improvement that is actually related to the training program.
    • Supervisors – of participants estimate the impact
      of training on the output variables. Present supervisors with the total amount
      of improvement, and ask them to indicate the percent related to training.
    • Senior Managers – estimate the impact of training
      by providing an estimate or adjustment to reflect the portion of the improvement
      related to the training program. While perhaps inaccurate, having senior management
      involved in this process develops ownership of the value and buy-in process.
    • Experts –estimate the impact of training on the performance
      variable. Because these estimates are based on previous experience, experts
      must be familiar with the type of training and the specific situation.

    Customers sometimes provide input on the extent to which training has influenced
    their decision to use a product or service. Although this approach has limited
    applications, it can be quite useful in customer service and sales training.

    Converting Data to Monetary Values

    A number of techniques are available to convert data to monetary values; the
    selection depends on the type of data and the situation.

    • Convert output data to profit contribution or cost savings. With this technique,
      output increases are converted to monetary value based on their unit contribution
      to profit or the unit of cost reduction. These values are readily available
      in most organizations and are seen as generally accepted standard values.
    • Calculate the cost of quality, and covert quality improvements directly
      to cost savings. This standard value is available in many organizations for
      the most common quality measures (such as rejects, rework, and scrap).
    • Use the participants’ wages and employee benefits as the value for
      time in programs where employee time is saved. Because a variety of programs
      focus on improving the time required to complete projects, processes, or daily
      activities, the value of time becomes an important and necessary issue. The
      use of total compensation per hour provides a conservative estimate for the
      value of time.
    • Use historical costs when they are available for a specific variable. In
      this case, use organizational cost data to establish the specific value of
      an improvement.
    • Use internal and external experts, when available, to estimate a value for
      an improvement. In this situation, the credibility of the estimate hinges
      on the expertise and reputation of the individual.
    • Use external databases, when available, to estimate the value or cost of
      data items. Research, government, and industry databases can provide important
      for these values. The difficulty lies in finding a specific database related
      to the situation.
    • Ask participants to estimate the value of the data item. For this approach
      to be effective, participants must understand the process and be capable of
      providing a value for the improvement.
    • Require supervisors and managers to provide estimates when they are willing
      and capable of assigning values to the improvement. This approach is especially
      useful when participants are not fully capable of providing this input or
      in situations where supervisors or managers need to confirm or adjust the
      participant’s estimate.

    Converting data to monetary value is very important in the ROI model and is
    absolutely necessary to determine the monetary benefits from a training program.
    The process is challenging, particularly with the conversion of soft data, but
    can be methodically accomplished using one or more of the above techniques.

    Tabulating Program Costs

    The other part of the equation in a cost/benefit analysis is the cost of the
    program. Tabulating the costs involves monitoring or developing all of the related
    costs of the program targeted for the ROI calculation. Include the following
    items among the cost components.

    • Cost to design and develop the program, possibly prorated over the expected
      life of the program
    • Cost of all program materials provided to each participant
    • Cost for the instructor/facilitator, including preparation time as well
      as delivery time.
    • Cost of the facilities for the training program.
    • Cost of travel, lodging and meals for the participants, if applicable.
    • Salaries, plus employee benefits of the training function, allocated in
      some convenient way.

    In addition, specific cost related to the needs assessment and evaluation should
    be included, if appropriate. The conservative approach is to include all of
    these costs so that the total is fully loaded.

    Calculating the ROI

    Calculate the ROI using the program benefits and costs. The BCR is the program
    benefits divided by costs:

    • BCR = program benefits / program costs
    • (Sometimes this ratio is stated as a cost/benefit ratio, although the formula
      is the same as BCR).

    The net benefits are the program benefits minus the costs:

    • Net benefits = program benefits – program costs

    The ROI uses the net benefits divided by programs costs:

    • ROI (%) = net benefits / program costs x 100

    Use the same basic formula in evaluating other investments where the ROI is
    traditionally reported as earnings divided by investment. The ROI from some
    training programs is high. For example, in sales training, supervisory training,
    and managerial training, the ROI can be quite large, frequently over 100 percent,
    while ROI value for technical and operator training may be lower.

    Additional Resources to Guide Evaluation of Your Training

    Evaluation of Training

    Training: There is no “Cookbook” Approach

    Evaluating Training and Technical Assistance
    the Bottom-Line Payoff for Training

    Approach to Evaluating Training


    Ensuring Behavior Change Occurs From Your Training

    to Measure Post Training Effectiveness

    ROA Formula / Return on Assets Calculation
    Determining Training ROI
    on Employer-Provided Formal Training Programs

    Talbut on Testing What You Teach

    Shonta Smith
    on Student Ratings of Instruction: Evaluating the Professor

    Hill-Carter on Looking at Assessment in a Different Light

    Assessments: Personality Counts

    Donald L Kirkpatrick’s training evaluation model
    – the four levels of learning evaluation

    Brainstorm: Evaluating Trainers

    Is Lecture Learning?
    Seven Steps To Guarantee Great Training Results
    Four Reasons for a Needs Assessment Survey Plus Two

    5 Ways to Assess Training Results
    Addressing the Dreams–Making Training More Effective
    On Evaluating Standardized Tests

    Evaluating Online Learning

    Evaluating Online Learning
    What Are the Signs of a High-Quality Online Course?

    Learn More in the Library’s Blogs Related to This

    In addition to the articles on this current page, also see the following blogs
    that have posts related to this topic. Scan down the blog’s page to see various
    posts. Also see the section “Recent Blog Posts” in the sidebar of
    the blog or click on “next” near the bottom of a post in the blog.
    The blog also links to numerous free related resources.

    Library’s Career
    Management Blog

    Library’s Human Resources

    Library’s Leadership Blog
    Library’s Supervision

    Training and Development Blog

    Go to main Training
    and Development

    For the Category of Training and Development:

    To round out your knowledge of this Library topic, you may
    want to review some related topics, available from the link below.
    Each of the related topics includes free, online resources.

    Also, scan the Recommended Books listed below. They have been
    selected for their relevance and highly practical nature.

    Related Library Topics

    Recommended Books