Free Nonprofit Micro-eMBA Module #11

Designing Your Program Evaluation Plans

© Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC.

Much of the information in this module is adapted from Field Guide to Nonprofit Program Design, Marketing and Evaluation.) This learning module is in the nonprofit organization development program. However, this module can also be used by anyone as a self-study exercise to learn more about nonprofit program evaluation.

Sections of This Module Include the Following

Introduction
Outcomes
Materials for Review
Suggested Topics for Reflection and Discussion
Activities to Build Systems and Practices
Assessments
Tracking Open Action Items



INTRODUCTION

Nonprofits deliver their services to the their communities in the form of programs. There is a common misconception that a program is the activities of delivering services -- but that's not correct. A program is

  • a highly integrated set of activities
  • that aims to meet a verified, unmet need in the community
  • by accomplishing certain outcomes
  • among certain groups clients and
  • by evaluating that those outcomes are being achieved among those clients.

Thus, the design of a program should include verifying that there indeed is an unmet need in the community. That has to include more than the passionate assertions of the founders of the nonprofit -- it has to include objective, verifiable evidence of the unmet need. The design should include verifying what outcomes need to be achieved, including what new knowledge, skills and abilities among the clients to be served. Program evaluation should regularly be conducted to verify that those outcomes are being achieved.

The activities of program evaluation can include a wide variety of methods to evaluate many aspects of programs in nonprofits. As with any evaluation activity, the type of evaluation used, depends on what the organization wants to be able to do as a result of the evaluation, whether it's, for example, to improve the program's processes or measure results of the program. There are many types of program evaluations. For example, a goals-based evaluation examines if the goals of a program were achieved or not. A process evaluation examines the strengths and weaknesses of the program's processes. An outcomes-evaluation examines what outcomes were achieved and the extent to which desired outcome were achieved.

There are numerous books and other materials that provide in-depth analysis of evaluations, their designs, methods, combination of methods and techniques of analysis. However, personnel do not have to be experts in these topics to carry out a useful program evaluation. The "20-80" rule applies here, that 20% of effort generates 80% of the needed results. It's better to do what might turn outs to be an average effort at evaluation than to do no evaluation at all because of concerns of not doing it perfectly. Far too many program evaluations become extended, scientifically-based, data-driven efforts that generate information that is either impractical or irrelevant -- if the information is understood at all. This learning module orients personnel to the nature of program evaluation and how it can be carried out in a realistic and practical fashion.

NOTE ABOUT BOARD COMMITTEES: Consider using the Board Programs and Marketing Committee to review and help guide implementation the information in this learning module. Major activities and goals from this learning module could be incorporated in that Committee's Committee Work Plan.



OUTCOMES

General Program Evaluation

  1. Understanding Program Evaluation, Goals and Types
  2. Gain Overview of Data Collection Methods
  3. Understand Guidelines for Analyzing, Interpreting and Reporting Information
  4. Recognize Pitfalls to Avoid
  5. Evaluate Your Program Planning and Evaluation Practices

Outcomes-Based Evaluations

  1. Get Ready for Your Outcomes Planning
  2. Select Your Outcomes and Target Outcomes
  3. Develop Your Logic Model
  4. Identify Indicators for Each of Your Target Outcomes
  5. Decide What Data You Need and How You Will Collect It
  6. Gain Guidelines for Your Data Analyses
  7. Draft Your Outcomes-Based Evaluation Report

MATERIALS FOR REVIEW

  • The following materials will help you address each of the topics and learning activities in this module.

Basic Guide to Program Evaluation (about evaluation, in general) -- read all of each of the following sections:
- - - Program Evaluation
- - - Where Program Evaluation is Helpful
- - - Basic Ingredients: Organization and Program(s)
- - - Planning Your Program Evaluation
- - - Some Major Types of Program Evaluation
- - - Overview of Methods to Collect Information
- - - Selecting Which Methods to Use
- - - Analyzing and Interpreting Information
- - - Reporting Evaluation Results
- - - Who Should Carry Out the Evaluation?
- - - Contents of an Evaluation Plan
- - - Pitfalls to Avoid

Outcomes Evaluation -- read all of each of the following sections:
- - - Reasons for Priority on Outcomes-Based Evaluation
- - - Basic Principles for Small Nonprofits to Remember Before Starting
- - - What is Outcomes-Based Evaluation?
- - - Common Myths to Get Out of the Way Before You Start Your Outcomes Planning
- - - Planning Any Type of Evaluation Includes Answers to These Very Basic Questions
- - - Planning Your Outcomes Evaluation -- Step 1: Getting Ready
- - - Planning Your Outcomes Evaluation -- Step 2: Choosing Outcomes
- - - Planning Your Outcomes Evaluation -- Step 3: Selecting Indicators
- - - Planning Your Outcomes Evaluation -- Step 4: Planning Data/Info Collection
- - - Planning Your Outcomes Evaluation -- Step 5: Piloting/Testing
- - - Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results



SUGGESTED TOPICS FOR REFLECTION AND DISCUSSION

  • Learners are strongly encouraged to discuss the following questions with peers, board members, management and staff, as appropriate.

Program Evaluation in General

1. What is "program evaluation"? Name at least three of its benefits -- or three situations when it is helpful? (See Program Evaluation and Where Program Evaluation is Helpful.)

2. What are the two key ingredients you need to have before you can conduct program evaluation? (HINT: The answer may be simpler than you think, but it's surprising how many nonprofit do not have both of these ingredients before they set out to conduct program evaluation!) (See Basic Ingredients.)

3. What determines how you will conduct your program evaluation? (HINT: Your program evaluation plans depend on what _____ you need to collect in order to make major_____.) (See Planning Your Program Evaluation.)

4. Name at least four of the seven key considerations when designing program evaluations. (See Planning Your Program Evaluation.)

5. What are three major types of evaluations typically done when carrying out nonprofit program evaluations? (There are many types of evaluations. This question asks about major types of evaluation used in program evaluation in nonprofits.) (See Some Major Types of Program Evaluation.)

6. Name at least four of the six major methods of data collection as listed in the information about program evaluation. (See Overview of Methods to Collect Information .)

7. Name at least five of the nine questions that should be considered when selecting the evaluation method (when conducting program evaluation). (See Selecting Which Methods to Use.)

8. What are the four levels of information that can be gathered from people during a program evaluation? (See Selecting Which Methods to Use.)

9. Name at least three of the five pitfalls of program evaluation planning. (See Pitfalls to Avoid.)

Outcomes-Based Evaluation

1. What are the major reasons that outcomes-based evaluation is a top priority among nonprofit funders and service providers today. (See Reasons for Priority on Implementing Outcomes-Based Evaluation.)

2. What are program inputs? Activities? Outputs? Outcomes? Outcome targets? Indicators? (See What is Outcomes-Based Evaluation? )

3. What is the difference between outputs and outcomes? Outputs and indicators? Indicators and outcome targets? (See What is Outcomes-Based Evaluation? )

4. Name at least three of the six myths listed in the suggested materials for review. (See Common Myths to Get Out of the Way Before You Start Planning)

5. Name at least five of the typical questions answered in almost any type of evaluation. (See Planning Any Type of Evaluation Includes Answers to These Very Basic Questions.)

6. Name the guidelines for picking which program to evaluate. (See Planning Your Outcomes Evaluation -- Step 1: Getting Ready.)

7. What is a logic model? (See Planning Your Outcomes Evaluation -- Step 2: Choosing Outcomes.)

8. What should you look for in short-term outcomes? Intermediate outcomes? Long-term outcomes? (See Planning Your Outcomes Evaluation -- Step 2: Choosing Outcomes.)

9. What is a questionnaire? Interview? Case study? (See Planning Your Outcomes Evaluation -- Step 4: Planning Data/Info Collection.)

10. What are some practical considerations when selecting which data collection method to use? (See Planning Your Outcomes Evaluation -- Step 4: Planning Data/Info Collection.)

11. What should you always do with the data that you've collected before you analyze it? (See Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results.)

12. What ultimately determines the level and scope of the evaluation report? (See Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results.)

13. Name at least five of the sections of a typical evaluation report. (See Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results.)



ACTIVITIES TO BUILD SYSTEMS AND PRACTICES

  • Learners are strongly encouraged to complete the following activities, and share and discuss results with peers, board members, management and staff, as appropriate.
  • As you proceed through the following activities, be sure to note any incomplete actions in the Action Item Planning List.

Writing Your Overall Program Evaluation Plan

Complete the following checklist about each of the program that you would like to evaluate.

Purpose of Evaluation?
What do you want to be able to decide as a result of the evaluation? For example:
__ Understand, verify or increase impact of products or services on customers/clients (eg, outcomes evaluation)
__ Improve delivery mechanisms to be more efficient and less costly (eg, process evaluation)
__ Verify that we're doing what we think we're doing (eg, process evaluation)
__ Clarify program goals, processes and outcomes for management planning
__ Public relations
__ Program comparisons, eg., to decide which should be retained
__ Fully examine and describe effective programs for duplication elsewhere
__ Other reason(s)

Audience(s) for the Evaluation?
Who are the audiences for the information from the evaluation, for example:
__ Clients/customers
__ Funders/Investors
__ Board members
__ Management
__ Staff/employees
__ Other(s) _________________________________________________

What Kinds of Information Are Needed?
What kinds of information are needed to make the decision you need to make and/or enlighten your intended audiences, for example, information to understand:
__ The process of the product or service delivery (its inputs, activities and outputs)
__ The customers/clients who experience the product or service
__ Strengths and weaknesses of the product or service
__ Benefits to customers/clients (outcomes)
__ How the product or service failed and why, etc.
__ Other type(s) of information?

Type of Evaluation?
Based on the purpose of the evaluation and the kinds of information needed, what types of evaluation is being planned?
__ Goal-based?
__ Process-based?
__ Outcomes-based?
__ Other(s)? ___________________________________________________

Where Should Information Be Collected From?
__ Staff/employees
__ Clients/customers
__ Program documentation
__ Funders/Investors
__ Other(s) ________________________________________________

How Can Information Be Collected in Reasonable and Realistic Fashion?
__ Questionnaires
__ Interviews
__ Documentation
__ Observing clients/customers
__ Observing staff/employees
__ Conducting focus groups among_____________________________________
__ Other(s)

When is the Information Needed?
________________________________________________________________

________________________________________________________________

What Resources Are Available to Collect the Information?

________________________________________________________________

________________________________________________________________

Writing Your Outcomes-Based Evaluation Plan

Complete each of the steps specified in the following sections. You will be asked to include some of your answers in the Framework for a Basic Outcomes-Based Evaluation Plan, which will culminate in your developing your basic outcomes-based evaluation plan. (You might want to convert the Framework to your preferred word-processing software to make it easier to modify.)

1. Planning Your Outcomes Evaluation -- Step 1: Getting Ready

2. Planning Your Outcomes Evaluation -- Step 2: Choosing Outcomes

3. Planning Your Outcomes Evaluation -- Step 3: Selecting Indicators

4. Planning Your Outcomes Evaluation -- Step 4: Planning Data/Info Collection

5. Planning Your Outcomes Evaluation -- Step 5: Piloting/Testing

6. Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results



ASSESSMENTS

1. Planning Indicators (Best Practices) (includes program planning and evaluation)



REMINDERS FOR THOSE IN THE ON-LINE DEVELOPMENT PROGRAM

1. Are you exchanging feedback with others about what you're learning in this program?

2. Are you sticking to your study schedule for this program?

3. Are you practicing your basic skills in management and leadership, including in problem solving and decision making, planning and meeting management?

4. Are you communicating throughout your organization by using your skills in internal communications?

5. Are you managing yourself? How many hours a week are you working? Are you noticing any signs of stress? If so, what are you doing about it?

6. One of the ways you might be able to tell if you're stressed out and/or losing perspective might be whether you're tracking details or not. Are you using the action item list referenced above?

7. Are you reflecting on learnings from past modules and how they build on the learning in this module? For example, are you seeing your organization from a systems view, as explained in the module "Starting and Understanding Your Nonprofit?"



TRACKING OPEN ACTION ITEMS

1. One of the first indicators that an organization is struggling is that open action items are not tracked and reviewed. (Open action items are required actions that have not yet been completed.) Instead, organization members only see and react to the latest "fires in the workplace". Whether open action items are critical to address now or not, they should not entirely be forgotten. Therefore, update and regularly review a list of open action items that includes listing each open action item, who is responsible to complete it, when it should be completed and any associated comments. When updating the list, consider action items as identified during discussions, learning activities and assessments in this module. Share and regularly review this action item list with the appropriate board, management and staff members in your nonprofit. You can use the following Action Item Planning List.

2. If you have questions, consider posing them in the national, free, online discussion group HRNET, which is attended by many human resource and organization development experts.


(Learners in the nonprofit organization development program can return to the nonprofit organization development program.)


Submit a link


For the Category of Evaluations (Many Kinds):

To round out your knowledge of this Library topic, you may want to review some related topics, available from the link below. Each of the related topics includes free, online resources.

Also, scan the Recommended Books listed below. They have been selected for their relevance and highly practical nature.

Related Library Topics

Recommended Books

Evaluation (General)

Program Evaluation



Evaluation (General)

The following books are recommended because of their highly practical nature and often because they include a wide range of information about this Library topic. To get more information about each book, just click on the image of the book. Also, a "bubble" of information might be displayed. You can click on the title of the book in that bubble to get more information, too.

Program Evaluation

The following books are recommended because of their highly practical nature and often because they include a wide range of information about this Library topic. To get more information about each book, just click on the image of the book. Also, a "bubble" of information might be displayed. You can click on the title of the book in that bubble to get more information, too.

Nonprofit Program Design, Marketing and Evaluation - Book Cover Field Guide to Nonprofit Program Design, Marketing and Evaluation
by Carter McNamara, published by Authenticity Consulting, LLC. There are few books, if any, that explain how to carefully plan, organize, develop and evaluate a nonprofit program. Also, too many books completely separate the highly integrated activities of planning, marketing and evaluating programs. This book integrates all three into a comprehensive, straightforward approach that anyone can follow in order to provide high-quality programs with strong appeal to funders. Includes many online forms that can be downloaded. Many materials in this Library topic are adapted from this book.

Also see

Business Research -- Recommended Books

Supervision (Evaluating Employees) -- Recommended Books




Find a Topic