Designing Your Program Evaluation Plans

Sections of this topic

    Free Nonprofit Micro-eMBA Module #11: Designing Your Program Evaluation Plans

    © Copyright Carter
    McNamara, MBA, PhD, Authenticity Consulting, LLC
    .

    Much of the information in this module is adapted from Field Guide to Nonprofit Program Design, Marketing
    , and Evaluation.
    ) This learning module is in the nonprofit organization development program.
    However, this module can also be used by anyone as a self-study
    exercise to learn more about nonprofit program evaluation.

    Sections of This Module Include the Following

    Introduction
    Outcomes
    Materials for Review
    Suggested Topics for Reflection and Discussion
    Activities to Build Systems and Practices
    Assessments
    Tracking Open Action Items


    INTRODUCTION

    Nonprofits deliver their services to their communities
    in the form of programs. There is a common misconception
    that a program is an activity of delivering services — but
    that’s not correct. A program is

    • a highly integrated set of activities
    • that aims to meet a verified, unmet need in the community
    • by accomplishing certain outcomes
    • among certain groups of clients and
    • by evaluating that those outcomes are being achieved among
      those clients.

    Thus, the design of a program should include verifying that
    there indeed is an unmet need in the community. That has to include
    more than the passionate assertions of the founders of the nonprofit
    — it has to include objective, verifiable evidence of the unmet
    need. The design should include verifying what outcomes need to
    be achieved, including what new knowledge, skills, and abilities
    among the clients to be served. Program evaluation should regularly
    be conducted to verify that those outcomes are being achieved.

    The activities of program evaluation can include a wide
    variety of methods to evaluate many aspects of programs in nonprofits.
    As with any evaluation activity, the type of evaluation used
    depends on what the organization wants to be able to do as a result
    of the evaluation, whether it’s, for example, to improve the program’s
    processes or measure the results of the program. There are many types
    of program evaluations. For example, a goals-based evaluation
    examines if the goals of a program were achieved or not. A process
    evaluation
    examines the strengths and weaknesses of the program’s
    processes. An outcomes evaluation examines what outcomes
    were achieved and the extent to which desired outcomes were achieved.

    There are numerous books and other materials that provide in-depth
    analysis of evaluations, their designs, methods, a combination of
    methods, and techniques of analysis. However, personnel do not
    have to be experts in these topics to carry out a useful program
    evaluation. The “20-80” rule applies here, that 20%
    of effort generates 80% of the needed results. It’s better to
    do what might turnouts be an average effort at evaluation
    than to do no evaluation at all because of concerns about not doing
    it perfectly. Far too many program evaluations become extended,
    scientifically-based, data-driven efforts that generate information
    that is either impractical or irrelevant — if the information
    is understood at all. This learning module orients personnel to
    the nature of program evaluation and how it can be carried out
    in a realistic and practical fashion.

    NOTE ABOUT BOARD COMMITTEES: Consider using the Board Programs
    and Marketing Committee to review and help guide implementation
    the information in this learning module. Major activities and
    goals from this learning module could be incorporated into that
    Committee’s Committee Work Plan.


    OUTCOMES

    General Program Evaluation

    1. Understanding Program Evaluation, Goals and Types
    2. Gain Overview of Data Collection Methods
    3. Understand Guidelines for Analyzing, Interpreting and Reporting
      Information
    4. Recognize Pitfalls to Avoid
    5. Evaluate Your Program Planning and Evaluation Practices

    Outcomes-Based Evaluations

    1. Get Ready for Your Outcomes Planning
    2. Select Your Outcomes and Target Outcomes
    3. Develop Your Logic Model
    4. Identify Indicators for Each of Your Target Outcomes
    5. Decide What Data You Need and How You Will Collect It
    6. Gain Guidelines for Your Data Analyses
    7. Draft Your Outcomes-Based Evaluation Report

    MATERIALS FOR REVIEW

    • The following materials will help you address each of
      the topics and learning activities in this module.

    Basic
    Guide to Program Evaluation (about evaluation, in general)

    — read all of each of the following sections:
    – – – Program
    Evaluation

    – – – Where
    Program Evaluation is Helpful

    – – – Basic
    Ingredients: Organization and Program(s)

    – – – Planning
    Your Program Evaluation

    – – – Some
    Major Types of Program Evaluation

    – – – Overview
    of Methods to Collect Information

    – – – Selecting
    Which Methods to Use

    – – – Analyzing
    and Interpreting Information

    – – – Reporting
    Evaluation Results

    – – – Who
    Should Carry Out the Evaluation?

    – – – Contents
    of an Evaluation Plan

    – – – Pitfalls
    to Avoid

    Outcomes Evaluation — read all of each of the following sections:
    – – – Reasons
    for Priority on Outcomes-Based Evaluation

    – – – Basic
    Principles for Small Nonprofits to Remember Before Starting

    – – – What
    is Outcomes-Based Evaluation?

    – – – Common
    Myths to Get Out of the Way Before You Start Your Outcomes Planning

    – – – Planning
    Any Type of Evaluation Includes Answers to These Very Basic Questions

    – – – Planning
    Your Outcomes Evaluation — Step 1: Getting Ready

    – – – Planning
    Your Outcomes Evaluation — Step 2: Choosing Outcomes

    – – – Planning
    Your Outcomes Evaluation — Step 3: Selecting Indicators

    – – – Planning
    Your Outcomes Evaluation — Step 4: Planning Data/Info Collection

    – – – Planning
    Your Outcomes Evaluation — Step 5: Piloting/Testing

    – – – Planning
    Your Outcomes Evaluation — Step 6: Analyzing/Reporting Results


    SUGGESTED TOPICS FOR REFLECTION
    AND DISCUSSION

    • Learners are strongly encouraged to discuss the following
      questions with peers, board members, management and staff, as
      appropriate.

    Program Evaluation in General

    1. What is “program evaluation”? Name at least three
    of its benefits — or three situations when it is helpful? (See
    Program Evaluation and Where Program Evaluation is Helpful.)

    2. What are the two key ingredients you need to have before
    you can conduct program evaluation? (HINT: The answer may be simpler
    than you think, but it’s surprising how many nonprofit do not
    have both of these ingredients before they set out to conduct
    program evaluation!) (See Basic Ingredients.)

    3. What determines how you will conduct your program evaluation?
    (HINT: Your program evaluation plans depend on what _____ you
    need to collect in order to make major_____.) (See Planning Your Program Evaluation.)

    4. Name at least four of the seven key considerations when
    designing program evaluations. (See Planning Your Program Evaluation.)

    5. What are three major types of evaluations typically done
    when carrying out nonprofit program evaluations? (There are many
    types of evaluations. This question asks about major types of
    evaluation used in program evaluation in nonprofits.) (See Some Major Types of Program Evaluation.)

    6. Name at least four of the six major methods of data collection
    as listed in the information about program evaluation. (See Overview of Methods to Collect Information .)

    7. Name at least five of the nine questions that should be
    considered when selecting the evaluation method (when conducting
    program evaluation). (See Selecting Which Methods to Use.)

    8. What are the four levels of information that can be gathered
    from people during a program evaluation? (See Selecting Which Methods to Use.)

    9. Name at least three of the five pitfalls of program evaluation
    planning. (See Pitfalls to Avoid.)

    Outcomes-Based Evaluation

    1. What are the major reasons that outcomes-based evaluation
    is a top priority among nonprofit funders and service providers
    today. (See Reasons for Priority on Implementing Outcomes-Based
    Evaluation
    .)

    2. What are program inputs? Activities? Outputs? Outcomes?
    Outcome targets? Indicators? (See What is Outcomes-Based Evaluation? )

    3. What is the difference between outputs and outcomes? Outputs
    and indicators? Indicators and outcome targets? (See What is Outcomes-Based Evaluation? )

    4. Name at least three of the six myths listed in the suggested
    materials for review. (See Common Myths to Get Out of the Way Before You
    Start Planning
    )

    5. Name at least five of the typical questions answered in
    almost any type of evaluation. (See Planning Any Type of Evaluation Includes Answers
    to These Very Basic Questions
    .)

    6. Name the guidelines for picking which program to evaluate.
    (See Planning Your Outcomes Evaluation — Step 1:
    Getting Ready.
    )

    7. What is a logic model? (See Planning Your Outcomes Evaluation — Step 2:
    Choosing Outcomes
    .)

    8. What should you look for in short-term outcomes? Intermediate
    outcomes? Long-term outcomes? (See Planning Your Outcomes Evaluation — Step 2:
    Choosing Outcomes
    .)

    9. What is a questionnaire? Interview? Case study? (See Planning Your Outcomes Evaluation — Step 4:
    Planning Data/Info Collection
    .)

    10. What are some practical considerations when selecting which
    data collection method to use? (See Planning Your Outcomes Evaluation — Step 4:
    Planning Data/Info Collection
    .)

    11. What should you always do with the data that you’ve
    collected before you analyze it? (See Planning Your Outcomes Evaluation — Step 6:
    Analyzing/Reporting Results
    .)

    12. What ultimately determines the level and scope of the evaluation
    report? (See Planning Your Outcomes Evaluation — Step 6:
    Analyzing/Reporting Results
    .)

    13. Name at least five of the sections of a typical evaluation
    report. (See Planning Your Outcomes Evaluation — Step 6:
    Analyzing/Reporting Results
    .)


    ACTIVITIES TO BUILD SYSTEMS AND
    PRACTICES

    • Learners are strongly encouraged to complete the following
      activities, and share and discuss results with peers, board members,
      management and staff, as appropriate.
    • As you proceed through the following activities, be sure
      to note any incomplete actions in the Action Item Planning List.

    Writing Your Overall Program Evaluation Plan

    Complete the following checklist about each of the program
    that you would like to evaluate.

    Purpose of Evaluation?
    What do you want to be able to decide as a result of the evaluation?
    For example:
    __ Understand, verify or increase impact of products or services
    on customers/clients (eg, outcomes evaluation)
    __ Improve delivery mechanisms to be more efficient and less costly
    (eg, process evaluation)
    __ Verify that we’re doing what we think we’re doing (eg, process
    evaluation)
    __ Clarify program goals, processes and outcomes for management
    planning
    __ Public relations
    __ Program comparisons, eg., to decide which should be retained
    __ Fully examine and describe effective programs for duplication
    elsewhere
    __ Other reason(s)

    Audience(s) for the Evaluation?
    Who are the audiences for the information from the evaluation,
    for example:
    __ Clients/customers
    __ Funders/Investors
    __ Board members
    __ Management
    __ Staff/employees
    __ Other(s) _______________

    What Kinds of Information Are Needed?
    What kinds of information are needed to make the decision you
    need to make and/or enlighten your intended audiences, for example,
    information to understand:
    __ The process of the product or service delivery (its inputs,
    activities and outputs)
    __ The customers/clients who experience the product or service
    __ Strengths and weaknesses of the product or service
    __ Benefits to customers/clients (outcomes)
    __ How the product or service failed and why, etc.
    __ Other type(s) of information?

    Type of Evaluation?
    Based on the purpose of the evaluation and the kinds of information
    needed, what types of evaluation is being planned?
    __ Goal-based?
    __ Process-based?
    __ Outcomes-based?
    __ Other(s)? ___________________________________________________

    Where Should Information Be Collected From?
    __ Staff/employees
    __ Clients/customers
    __ Program documentation
    __ Funders/Investors
    __ Other(s) _____________

    How Can Information Be Collected in Reasonable and Realistic
    Fashion?

    __ Questionnaires
    __ Interviews
    __ Documentation
    __ Observing clients/customers
    __ Observing staff/employees
    __ Conducting focus groups among ______

    When is the Information Needed?

    What Resources Are Available to Collect the Information?

    Writing Your Outcomes-Based Evaluation Plan

    Complete each of the steps specified in the following sections.
    You will be asked to include some of your answers in the Framework for a Basic Outcomes-Based Evaluation
    Plan
    , which will culminate in your developing your basic outcomes-based
    evaluation plan. (You might want to convert the Framework to your
    preferred word-processing software to make it easier to modify.)

    1. Planning Your Outcomes Evaluation — Step 1:
    Getting Ready

    2. Planning Your Outcomes Evaluation — Step 2:
    Choosing Outcomes

    3. Planning Your Outcomes Evaluation — Step 3:
    Selecting Indicators

    4. Planning Your Outcomes Evaluation — Step 4:
    Planning Data/Info Collection

    5. Planning Your Outcomes Evaluation — Step 5:
    Piloting/Testing

    6. Planning Your Outcomes Evaluation — Step 6:
    Analyzing/Reporting Results


    ASSESSMENTS

    1. Planning
    Indicators (Best Practices) (includes program planning and evaluation)


    REMINDERS FOR THOSE IN THE ON-LINE DEVELOPMENT PROGRAM

    1. Are you exchanging feedback with others about what you’re
    learning in this program?

    2. Are you sticking to your study schedule for this program?

    3. Are you practicing your basic skills in management and leadership,
    including in problem solving and decision making, planning and
    meeting management?

    4. Are you communicating throughout your organization by using
    your skills in internal communications?

    5. Are you managing yourself? How many hours a week are you
    working? Are you noticing any signs of stress? If so, what are
    you doing about it?

    6. One of the ways you might be able to tell if you’re stressed
    out and/or losing perspective might be whether you’re tracking
    details or not. Are you using the action item list referenced
    above?

    7. Are you reflecting on learnings from past modules and how they build on the learning
    in this module? For example, are you seeing your organization from a systems view,
    as explained in the module “Starting and Understanding Your Nonprofit?”


    TRACKING OPEN ACTION ITEMS

    1. One of the first indicators that an organization is struggling
    is that open action items are not tracked and reviewed. (Open
    action items are required actions that have not yet been completed.)
    Instead, organization members only see and react to the latest
    “fires in the workplace”. Whether open action items
    are critical to address now or not, they should not entirely be
    forgotten. Therefore, update and regularly review a list of open
    action items that includes listing each open action item, who
    is responsible to complete it, when it should be completed and
    any associated comments. When updating the list, consider action
    items as identified during discussions, learning activities and
    assessments in this module. Share and regularly review this action
    item list with the appropriate board, management and staff members
    in your nonprofit. You can use the following Action Item Planning List.

    2. If you have questions, consider posing them in the national,
    free, online discussion group hr.com, which is attended
    by many human resource and organization development experts.


    (Learners in the nonprofit organization development
    program can return to the nonprofit organization development program.)


    For the Category of Evaluations (Many Kinds):

    To round out your knowledge of this Library topic, you may
    want to review some related topics, available from the link below.
    Each of the related topics includes free, online resources.

    Also, scan the Recommended Books listed below. They have been
    selected for their relevance and highly practical nature.

    Related Library Topics

    Recommended Books