Effective interviewing is both an art and a science and takes training and lots of practice!  As I mentioned in my previous post, interviewers need to think quickly on their feet.  This is important because they need to “go with the flow” so it seems more like a friendly conversation than an interrogation and still manage to cover all the major questions.   If resources are very tight, consider networking to find a highly motivated graduate student or professional who is a stay-at-home parent who may be willing to conduct and analyze phone interviews from home:

• either in exchange for a modest stipend or
• pro bono to stay in touch with their field or branch out into a new area of expertise

In my early days of beginning to learn about qualitative methods (e.g. interviews, focus groups), I received a lot of help from Earl Babbie’s surprisingly down-to-earth textbook “The Practice of Social Research.” I still recommend this book as a great resource for interviewers.

Tip #4: Obtain audio recordings with the permission of your respondents.

Remember to make a decent audio recording of interviews and to obtain consent before doing so.  Some evaluators decide to translate interviews word-for-word, even including non-verbal events such as pauses, laughter, etc.  Weigh the benefits and costs of transcription versus detailed notes.

Tip #5: Conduct interviews and begin analyzing results simultaneously

Keep an eye out for emerging patterns and themes as you conduct interviews.  Analyzing the interview results for themes, also known as qualitative analysis or coding, as you go, helps alert you to modifications you may need to make to questions to best capture the information you need.  For example, in a recent set of interviews, I soon realized that my questions needed to be more direct.  For example, “do you receive XYZ type of support?” versus the more indirect: what benefits do you receive from this program?  (Although the indirect question can yield a wealth of information on unanticipated outcomes that are still very important!)

Tip #6: Allocate time to learn to use software that assists with qualitative analysis of your data, if needed.

Generally, these tools are very beneficial if you are using multiple interviewers and have a large number of respondents, or if the analyst detests the tedious work that accompanies coding transcripts for themes.  These tools do not replace the important role of manually reading and re-reading the interview transcripts though.  The American Evaluation Association has a LinkedIn group that is a great resource for questions related to which software to use and for those interested in evaluation in general!  http://www.linkedin.com/

Tip #7:  Begin writing the interview report even before you have finished analyzing all the data

Stepping back and beginning to see the big picture helps to deal with analysis paralysis or the condition of over-thinking about and over-analyzing the data.  Your inputs affect your outputs.  Well-designed interview questions that are based on the peer-reviewed literature as well as candid feedback from your program stakeholders, in concert with skillful interviewing techniques, all contribute to facilitating a meaningful and informative interview report.

_____________________________________________________________________________________________________________________

Priya Small has extensive experience in collaborative evaluation planning, instrument design, data collection, grant writing and facilitation.  Contact her at priyasusansmall@gmail.com.  Visit her website at http://www.priyasmall.wordpress.com.  See her profile at http://www.linkedin.com/in/priyasmall/

In the original article I talked about how when you develop a strategic plan, it should energize staff and board members to be more involved in fundraising. The plan and the planning process should enable you to communicate clearly with focus on your mission, your goals and your priorities. The strategic plan helps you find and organize the words that lead to action and results. This is an important output for any strategic plan.

A comment made by Dan Clark startled me at first. He said “Your post helps a nonprofit see the value of connecting their fund raising efforts to their mission.” My spotlight was on how doing a strategic plan could benefit your fundraising effort. But Dan reminds us that even if you aren’t doing a strategic plan make sure you are connecting your fund raising efforts with your mission.

Don’t all nonprofits know that they need to connect their fund raising efforts to their mission? The answer is pretty much yes to that question but more lukewarm if asked how well they do it. Many nonprofits just don’t communicate why they are worthy of a donor’s gift very well. Your strategic plan should help you identify your strengths and opportunities. The planning exercises will help you frame stronger key messages. Having fundraising be an issue to be addressed in your strategic planning will make sure that communications about your mission, goals and specific plans are a key output item from your plan.

Simone Joyaux and Marilyn Donnellan commented –each with their own framing – that the strategic plan should be holistic, encompassing all facets of the organization. I certainly agree but since my article had such a singular focus on fundraising I didn’t make that clear at all.

Finally Robert Hodge offered a fresh idea for a strategic plan. He recommends a two section approach – one for short term and one for long term. The short term plan addresses how limited resources are to be strategically allocated. The long term plan identifies the steps, priorities and additional resources needed to achieve them. The long range plan becomes the basis for fund raising apart from that of the annual fund. I was not familiar with this approach, but I can see how effective it can be in integrating your strategic plan and fundraising. Certainly, this is a good approach for an organization with very limited resources to come to terms with focusing on its priorities and having a plan that requires resource development for the next steps.

When I work with an organization with a strategic plan I am always conscious that we will fail if we produce a nice binder or report that is filed away never to be looked at again. You want to have a report that becomes dog eared with frequent use. Many nonprofits have done belt tightening due to the economy in the last few years. They are ready now, to take stock of where they are, define where they want to be and develop a plan for what they have to do to get to a healthier place. A lot is changing for nonprofits in the way we do business and expectations of us. Going through a strategic planning process, putting it all out on the table, deciding what your priorities are and recommitting to ACT on the priorities is a good first step to recovery. And yes, it is about a great deal more than fundraising.

Tip #1:  Decide beforehand whether interviews are the most effective and efficient way of collecting the data you need.

Weigh the pros and cons of interviews:

Pros of interviews:

Interviews may:

• Provide opportunities to probe for information that you may not otherwise think to ask for in a written questionnaire.
• Give you information and stories that people may not otherwise share in a written survey.
• Help you build rapport with interviewees and help identify stakeholders who really care about the program and may want to get further involved in the evaluation.  Involving stakeholders is key to a successful evaluation.  (see my previous post on the CDC program evaluation model.)
• Help explain trends in quantitative data, explaining questions such as “why” and “how.”  They can give you a good idea of how programs work and can help you generate a program description that is critical for every evaluation.  Interviews can provide rich data that paint a picturesque portrait of your program.
• Have potential to facilitate the expression of opinions and feelings in the interviewees’ unique “voices.” They are a rich source of quotes for future grant proposals.
• Phone interviews are less expensive than in-person interviews.

Cons of interviews

• More resource-intensive: it is time consuming to conduct and participate in interviews, to transcribe them and to analyze data.
• Requires interviewers to be trained (again more resource-intensive—think training time, planning and designing training materials and presentations)
• Interviewers need to be articulate and to be able to think quickly “on their feet” and simultaneously think ahead to decide on the next question they need to ask, listen and take notes at the same time¹.
• Usually smaller samples are used: so the representativeness of your data is much more limited.  For example, your data only represents those 14 people interviewed versus representing 140 surveyed or possibly being able to infer results to a larger population when using a questionnaire.

In the end, if you decide that you really need the type of data that interviews provide, interviews can be really worth the extra time and effort!

Tip #2:  Carefully design and follow an interview script even if you are the only interviewer, and train interviewers.  Make sure the script and the training facilitate the following practices among interviewers:

1. upholding ethical standards of behavior,
2. building rapport and
3. safeguarding the quality of data.  Selected examples:

Adhere to ethical procedures such as informed consent

It can be so tempting to improvise, thinking that this will make the questions sound less rehearsed.  But this makes it really easy to forget important steps like informing participants of the purpose of the interviews and asking them whether they are interested in participating in the interview (informed consent).  Inform participants of potential risks and benefits of participating in the evaluation.  This is especially important when collecting highly confidential health-related data.

Some participants may give you reasons for why they cannot participate in the interviews. This is where the interviewer has to first carefully discern whether the interviewee is actually interested in participating. Do not assume that everyone has the time or interest in participating. The interviewer has to then strike the careful balance between addressing any barriers that would prevent an interested interviewee from participating versus maintaining a high standard of professional ethics by being respectful of the individual’s decision not to participate and being careful about statements that may be misperceived as pressuring or coercing participant.

Do not use leading questions

Do not use leading questions, that is, questions or statements that can unconsciously influence the interviewees to give certain answers.  Example of a leading question: “What are the some of the challenges program participants face in getting to classes?”  Versus “Do participants face challenges in getting to classes?”

Avoid double-barreled questions

Be extra vigilant to avoid double-barreled questions, these can easily creep in especially while spontaneously asking probing questions.  Example of a double barreled question: “Do you either send cards or call your program participants?”  Answer:  “Yes.”  The problem is that these questions don’t help you figure out which of the two options is used.

Consider hiring a professional

Since there may be other considerations that go into upholding ethical conduct, building rapport, and safeguarding data quality, when doing a do-it-yourself evaluation, one option may be to collaborate with a professional evaluator to design the interview script and to train your interviewers.

¹Earl Babbie.  (2001).  The Practice of Social Research, 9^th edition.  Wadsworth/Thomson Learning

___________________________________________________________________________________________________________________

Priya Small has extensive experience in collaborative evaluation planning, instrument design, data collection, grant writing and facilitation.  Contact her at priyasusansmall@gmail.com.  Visit her website at http://www.priyasmall.wordpress.com.  See her profile at http://www.linkedin.com/in/priyasmall/

There are some differences in what is important to men and women when it comes to philanthropy but there is also a great deal of similarity. Although this report focuses on women, it provides excellent insight into what is important to both of them. Here are some highlights. First the motivators with a statistically significant difference between men and women:

                                                                                                           Women (%)      Men (%)
Moved at How Gift Can Make a Difference                             81.7                  70.9
Giving to an organization that is efficient                              80.5                  69.2
Give Back to Community                                                              78.2                  63.3
Volunteer for the Organization                                                  65.7                  49.8
Support Same Orgs./Causes Annually                                    59.5                  67.9
Set Example for Young People                                                  43.6                   25.1
Further Legacy of Others                                                             33.1                   16.4

The top two motivators for men and women are the same but there is a statistically significant difference in the importance between them. What comes next seems very important to me. Men are much more likely to continue to give to the same organization from year to year than women are. Women are more likely to continually be considering which organizations to support and are more likely to give to the organizations that they are engaged with. The data is clear about the need to keep women philanthropists actively involved in order to maintain their support. This should not be a surprise to us. Men are more likely to want to buy the same shirt in the same color from the same store year after year. A woman would never consider doing such a thing. Guess what – this same behavioral pattern holds true for philanthropy. I must admit I’ve never thought about that.

Here’s the list in descending order of importance of motivating factors that do not have a statistical difference between men and women:
Feel Financially Secure
Political/Philosophical Beliefs
Remedy Issues Affecting Me Personally
Give Spontaneously to Support a Need
Religious Beliefs
Being Asked
Business Interests

No surprises here.

Here are some other key findings that I found particularly noteworthy:

87% of high net worth women volunteered.

60% of gifts go to general operating funds (almost the same as for men)

Men are more likely to make a capital gift and women are more likely to support a particular program.

2/3 of both men and women report a family tradition of giving

So when it is a couple living together – married or not – who makes the decision about giving anyway? Here are the answers:

                                                              Percent Responding

                                                                Women (%)       Men (%)

I decide                                                      38.8                43.4
Joint Decisions                                        48.4                40.9
Separate decisions                                 10.6                 14.6

So even if this data made you smile about how men and women view decision making, in the case of both men and women well over 80% of the people think they are involved in philanthropic giving decisions when there are couples involved. That’s important to keep in mind – and not to focus solely on the interests of either party.

I recommend that you read the whole report.  Click  here  for the link to the full report.

Click here for  the link to my companion piece.

Here’s the links to earlier articles that I have writter on related subjects:

2010 High Net Worth Philanthropy Study and What It Means for Small Nonprofits  

The Wired Wealthy and Online Giving – A New Study Provides a Fresh Analysis

In the last post, we covered steps 1-3 of the CDC’s evaluation depicted below:

A Framework for Evaluation.

Source: Centers for Disease Control and Prevention (CDC), Office of the Associate Director for Program (OADPG)

Step 4:  Gather Credible Evidence

What is credible evidence? Let us back up and consider credibility from various perspectives–those of funders, agency staff and program participants.  Involving most important stakeholder groups throughout the evaluation process and being open to learn from their experiences will increase the credibility of the evidence.  Some program participants may not believe sources like government agencies and doctors who are traditionally viewed as credible sources in professional circles.

Your evidence is only as good as the tools you use to collect it.  Use high quality tools, i.e., questionnaires, interview guides, etc.  Pay attention to validity issues–for e.g. do the questions really measure what you think they are measuring?  At the very least, choose indicators based on a review of the literature. Indicators are items being measured such as knowledge levels, numbers of low birth weights, etc. that shed light on the health or social condition that your program is trying to change.

Ask experts to review your evaluation tools and then pilot test them among program participants.  In some cases, it may be particularly important to use an evaluation tool that has been tested for reliability, i.e., does the questionnaire yield consistent results each time it is used?  In these cases I recommend, if possible, using a tool that has been published in the peer-reviewed literature.  University libraries often allow visitors to use their databases and to access peer-reviewed journals online.  Other tips from the CDC–the CDC recommends:

• choosing indicators wisely
• training staff in data collection,
• paying attention to data quality issues and
• protecting confidentiality of participants’ information

Step 5:  Justify Conclusions

All conclusions need to be based on evidence.  Take care also to base all your conclusions on sound statistical reasoning.  For e.g., one common mistake is to conclude that there is a cause and effect relationship on the basis of correlational data.  A statistical correlation only shows that two variables are associated with one another.  Take for example the following piece of evidence: depression is correlated with lower levels of perceived social support.  All we can conclude is that there is a correlation between depression and social support.  Lower levels of perceived social support could have contributed to the depression, or the depression itself could have led to social withdrawal, which then resulted in lower levels of perceived social support.  If you’re interested in a light and amusing read to familiarize yourself with such principles, I second evaluator John Gargani’s recommendation of  Darrell Huff’s classic book “How to Lie with Statistics.”

This is another step where it is important to continue engaging stakeholders. Encourage stakeholders to participate in the process of drawing conclusions from evidence.  This will increase their trust in the findings and will increase the chances that they will actually use the evaluation.

Step 6:  Ensure Use and Share Lessons Learned

So how do we ensure that evaluation findings are actually used? Like cooking, presentation is everything! People process visual information much more intuitively and naturally than verbal information.  Consider, for example, how well very young children respond to colors and pictures.  This principle translates into effectively communicating your findings to adult audiences as well. A hot topic in the field of evaluation is data visualization or how to display information using sound design principles.  While it is true that graphs can be confusing, effectively applying data visualization principles can produce graphs that are elegantly intuitive to a lay evaluation consumer.  For further study, read Edward Tufte’s classic book “The Visual Display of Quantitative Information.”

A tool that helps visually depict a variety of graphs and charts in one place is a data dashboard.  Think of it as a short cut to communicating information visually.  A dashboard is a display of multiple graphs all in the same location.   A resource for further reference is Stephen Few’s Information Dashboard Design: The Effective Visual Communication of Data.

To increase use of the evaluation findings, the CDC recommends:

• aligning evaluation design with how stakeholders plan to use the evaluation,
• translating findings into practical recommendations for the program and
• using reporting strategies that are customized to stakeholders’ specific needs.

For DIY (Do It Yourself) evaluators, I highly recommend renowned evaluation theorist Michael Quinn Patton’s book Utilization-Focused Evaluation.  You can listen to free recordings of two webinars by Michael Q. Patton here.  (You may need to download software first).  ”But I’m too busy managing my program to sit down and listen to webinars,” you protest.

I understand too well! Well, I listened to both webinars recently while doing housework and received very helpful guidance for a current project.

(To be continued)

____________________________________________________________________________________________________

Priya Small has extensive experience in collaborative evaluation planning, instrument design, data collection, grant writing and facilitation.  Contact her at priyasusansmall@gmail.com.  Visit her website at http://www.priyasmall.wordpress.com.  See her profile at http://www.linkedin.com/in/priyasmall/

This week-end I found myself navigating the underground tunnel system of a local university on my way to the library.  Although this was not my first time, it got me thinking of others.  If not for the signs, newer navigators would have either run into dead-ends or ended up walking in circles.   Evaluations can also go around in circles or run into dead-ends.  In this post I aim to whet your appetite for the evaluator’s version of signs and guideposts:  evaluation models or frameworks.

Some think of them as evaluation road maps or mental models.  Usually such models are based on years of experience and/or research.  Following such models will help to spare you costly mistakes.

Today I will briefly introduce the Centers for Disease Control and Prevention’s (CDC) Framework for Evaluation. A thorough presentation is beyond the scope of my post, so please review the references I have included for future study.

A Framework for Evaluation.

Source: Centers for Disease Control and Prevention (CDC), Office of the Associate Director for Program (OADPG)

Step 1:  Engage Stakeholders

Stakeholders include everyone linked to or benefiting from your program:  for e.g. participants, program staff, national staff, collaborators, funders and even evaluators.  Identify a small number of key stakeholders and involve them as much as possible throughout the lifespan of the evaluation.  Such involvement is crucial since it ensures that stakeholders, especially those belonging to vulnerable populations, are adequately represented.  A range of active and passive involvement strategies may include:

• forming an evaluation committee
• promoting engagement via
  • face to face meetings
  • capacity building activities
  • teleconferences
  • e-mail or discussion groups
  • simple interviews or surveys of stakeholders
  • letters and newsletters to inform them of evaluation activities and key decisions

The type of involvement strategy you choose should be custom-tailored to the specific needs of your particular program and stakeholders.  Pay close attention to organizational climate and of course, timing!

Step 2:  Describe the Program

Describing the program can be much harder than it deceptively seems!  Various stakeholders may have differing ideas of what the program entails or should entail.  Even an individual stakeholder’s perspectives can evolve over time. An iterative process is important to get everyone on the same page and to determine whether everyone’s intentions for the program reflect the actual program goals.

Once program goals are clarified, work backwards to develop a logic model, which is a flow chart demonstrating relationships between program components and the outcomes you are seeking.

Step 3:  Focus the Evaluation Design

Focused evaluations are the most useful.  Prioritize and focus your evaluation questions in collaboration with the small number of key stakeholders.   Consider how to best serve their needs and how to prioritize the competing needs of various stakeholders. Then choose the most appropriate evaluation methods that will provide you with the best answers to those evaluation questions.  Seek to balance:

• efficiency and practicality with
• the quality and type of data and the level of accuracy needed.



To be Continued…

Sources/Further References:

Centers for Disease Control and Prevention (CDC), Office of the Associate Director for Program (OADPG).  (2011) A Framework for Evaluation.  Retrieved February 6, 2012. From www.cdc.gov/eval/framework/index.htm A reliable, easy to navigate website hosted by the CDC.

Milstein, B., Wetterhall, S. and the CDC Evaluation Working Group. (2012)  A Framework for Program Evaluation: A Gateway to Tools.  The Community Toolbox. J. Nagy & S. B. Fawcett (Eds.). Retrieved February 6, 2012. From http://ctb.ku.edu/en/tablecontents/sub_section_main_1338.aspx The Community Tool Box is an online tutorial that is designed especially for community-based nonprofits and hosted by the University of Kansas.

________________________________________________________________________

Priya Small has extensive experience in collaborative evaluation planning, instrument design, data collection, grant writing and facilitation.  Contact her at priyasusansmall@gmail.com.  Visit her website at http://www.priyasmall.wordpress.com.  See her profile at http://www.linkedin.com/in/priyasmall/

I write and present about how small nonprofits should use social media.  I am usually writing about it from the standpoint of what a nonprofit should do.  This is about what Board Members should do.  Almost every small nonprofit I work with has an  underutilized web presence.  Some have made an investment which is big for them in a new website and are disappointed that the website does not generate much in the way of online donations.  I frequently wince when I see these websites and some of the fundamental  mistakes that have been made. 

But board members can do a great deal to boost the overall web presence of their nonprofit.  Today, in addition to a website, having a vibrant social media presence is an important tool in any nonprofit’s toolbox.  It is a key way to build relationships with existing supporters and to find new ones.

Having a website is not like “If we build it, they will come.”  They won’t – that only happens in the movies.  You have to DRIVE traffic to your website.  And the best way to do that is with a vibrant social media presence.

First let’s look at what a nonprofit should ideally be doing with its overall web presence:

• Engage existing supporters
• Find potential new friends (donors, volunteers, etc)
• Find volunteers
• Find people who are interested in attending events – and maybe sell tickets
• Identify potential Board Members
• Build better relationships with all of the above people
• Get donations

The last one – get donations – comes later.   First you have to do all the other things.  Board members can help with some of or all of these goals by just incorporating their nonprofit into what they already do with social media.  Different board members will use different aspects of social media and they can all contribute in their own way.  I am not talking about asking your friends to give money online.  If you feel comfortable doing that, by all means, go ahead. But that is not what I am suggesting here.  I am talking about how you can be an ambassador online just by tweaking – a very little bit – what you already do.  Let’s look at the possibilities.

Facebook

My article,  Demonstrating Board Leadership with Facebook  is getting record readership, and I am getting lots of feedback about sharing it with other Board members. I go into detail with these items in that article but here are the highlights for board members on Facebook:

1. Like the organization’s page
2. Like comments and comment on the page’s posts
3. Initiate posts on the organization’s wall.
4. Initiate posts on your own wall.
5. Post pictures

LinkedIn

LinkedIn has become an important resource for nonprofits.  You can use LinkedIn to give positive exposure to you and to your nonprofit. Here are a few ways:

1.      Update your profile

LinkedIn has a new category in its profile – Volunteer Experience & Causes.  You can enhance your profile and gives some exposure to your organization by updating your profile to include your Board Service.  You can see mine as an example.

Marion’s LinkedIn Profile

 2.      Donor and Board Member Prospecting

Use the search and advanced search to find people who have particular professions, went to your university and more to find people you know who may be a good match.  This takes a little time but you can do it whenever you have free time and you just might find some good prospects to invite to be a Board member or to invite to an event.

 3.      Update your Status

You can update your status to show activities you are involved with at your nonprofit – always include a link to their website.  Updating your status can serve to keep both your name and your nonprofit’s name in the sight of your business friends.  It is a win-win situation.  Short, but effective.  And if you have linked your LinkedIn and twitter accounts, you can just click to have this update appear automatically in twitter also.  Use this to promote events, seek new Board members, etc.

 Twitter

I love twitter and I am active on it.  I enjoy being part of an active nonprofit online community and learn about incredible opportunities, great resources and events all the time on twitter.  It has been wonderful to finally meet people in person at conferences or at a workshop who I have only known on twitter.  But I don’t see twitter as being very useful for the purpose of a Board Member using it to build relationships for a nonprofit.  If you are on twitter, be sure to mention your nonprofit sometimes especially if they are on twitter too.

There are two exceptions to this is that twitter is wonderful for  - advocacy and promoting events.  If advocacy is a part of what your nonprofit does, Board Members who are on twitter should definitely be using their 140 characters for advocacy.  Make sure you use relevant hastags (#) and you will see lots of new followers.

If your nonprofit is having an event and you have local followers promote the event and link to the nonprofit’s website – not the homepage but the event information page.  Be thoughtful in using hashtags so that more people than just your followers may see the tweets.

 Blogs

Do you read blogs related to the work of your nonprofit.  Be sure to leave comments and mention your organization in the comments when appropriate.

 Websites

Do you have a website – personal or small business.  Include your board membership in your biography and have a link to it in the bio.  If appropriate have a small “ad” for events that your nonprofit is having on your website.

Pinterest

Pinterest is new and you may not of heard of it, but it is very fast growing.  It is a new social media site that is perfect for people who like visuals and it is very simple and easy to use.  It is essentially a bunch of online bulletin boards.  Once you have a pinterest account – which you can associate with your facebook account, you can easily “pin” visuals on a board.  You can have lots of Boards and it is best to have all the visuals on that board be about a common topic.  This is the link to one of my favorites – nonprofit infographics by Beth Kanter:  http://pinterest.com/kanter/nonprofit-infographics/.  If you have a pinterest account, create some Boards with visuals about your nonprofit.  Cover an event or a program.  Use a brief description.  And then…post an invitation to see your Board on facebook, twitter LinkedIn, etc.  You get the idea – have it go viral.  I think that Pinterest may take off and become very popular so I am including it in this list of social media things to do.

I hope this article has given you something to think about.  These are all easy things to do but they can have a big impact if enough people take just a few steps.  And I promise it won’t seem like work at all, once you dive in, you’ll enjoy it.

Process Evaluations

These evaluate the program activities and methods a program uses to achieve its outcomes.  These activities should be directly linked to the intermediary and ultimate outcomes that your program will target. Examples of measures and evaluation questions include:

• number and demographics of participants served,
• number of activities such as number of prevention workshops conducted
• Were activities really implemented as planned?  How closely was curriculum followed, etc.

Impact Evaluations

These measure intermediary “outcomes” such as changes in knowledge, attitudes and behaviors that specifically link to the ultimate outcomes your program will target.  In order to be able to capture these changes, make sure to measure these items before (pre-test or baseline data) and after (post-test) your intervention.  For example, a heart disease prevention program may provide workshops targeting intermediary outcomes such as changes in knowledge, attitudes and behaviors related to nutrition and exercise.  We can view these intermediate outcomes as a “go-between” that connects the procedures with the outcomes.  A quick note: theory-driven and research-based program activities and measures are much more likely to actually produce/demonstrate the outcomes a program is seeking.

Outcome Evaluations

These evaluate changes in the ultimate outcomes your program is targeting.  Again, remember to collect this data before and after your intervention.  In our heart disease prevention program, we might measure changes in numbers of  coronary events such as heart attacks, etc.  In general, this level of outcomes can be harder to measure, especially in cases where stigma or shame is associated with the outcome you are measuring.

Process Evaluation ←→Impact Evaluation ←→ Outcome Evaluation

Thoughts

In program evaluation, both the means and the ends are equally critical.  Let us consider the importance of process evaluations since it is so easy to overlook the means.  The process indeed determines the outcome.  In a well-designed program, process measures link closely to intermediary outcomes, which in turn link closely to final outcomes.  If the process evaluation reveals shortages, that is, if the program has not really been implemented as planned, the final outcomes may suffer.   A good process evaluation provides an adequate program description over the course of the evaluation, which is so important!  A program description portrays what the program is essentially and really all about.  This is not that easy to accomplish but is worth the effort. What the program essentially is in its core will determine the outcomes it produces.

Different programmatic contexts call for different evaluations.  It is beyond the scope of this post to provide an exhaustive list of the different types of evaluations.  Here are a couple resources however:

http://www.cdc.gov/NCIPC/pub-res/dypw/03_stages.htm

Program Evaluation, Third Edition: Forms and Approaches (2006) by John M. Owen.

Question:

Evaluators, if forced to choose just two out of these 3 options, which would you rank as more important within your particular program context and why?

Announcement:

Who: The Center for Urban and Regional Affairs (CURA) at the University of Minnesota is offering

What: a two-day “Introduction to Program Evaluation” workshop by Stacey Stockdill, within its Spring Conference: Evaluation in a Complex World: Changing Expectations, Changing Realities

When:  Monday, March 26-Tuesday, March 27, 2012.

Where: University of Minnesota – Saint Paul Campus, Falcon Heights, MN 55113

Scholarships may be available for the Introduction to Program Evaluation workshop.  Scholarship application deadline:  February 24, 2012.

For more information: http://www.cura.umn.edu/news/scholarships-available-two-day-introduction-program-evaluation-workshop

Contact Person: William Craig

______________________________________________________________________________________

Priya Small has extensive experience in collaborative evaluation planning, instrument design, data collection, grant writing and facilitation.  Contact her at priyasusansmall@gmail.com.  Visit her website at http://www.priyasmall.wordpress.com.  See her profile at http://www.linkedin.com/in/priyasmall/

 This blog post is a companion piece to the resolutions post which has been very popular.   It is not intended to be a list of goals that you can cut and paste into your 2012 objectives.  Rather, it provides some guiding principles and things to think about when you are forming your specific goals for 2012.  At Marion Conway-Nonprofit Consultant  you can see the exact words of wisdom offered by each contributor.  The nonprofit expert contributors this year are Anne Ackerson, Susan Detweiler,  Jay Frost, Pamela Grow, Linda Lysakowski, Marc Pitman and Terrie Temkin.  You can visit their blog or website by clicking on their name.  I’ve organized the ideas into categories:

Fundraising/Development

ASK! – in capital and bold letters – often, and lots of ways – both traditional and using Social Media.  Don’t be shy about asking.

Get to know and connect with your donors in a personal way in all phases from cultivation through thank yous.

Get smarter – Try new approaches – Seek out training in marketing not just fundraising

Engage the CEO and Board in all phases

Think of the donor in terms of long term value and cultivate for the long term

Mission, Operations, Evaluation, Creativity, Boards

Understand and communicate your value proposition

Take evaluation seriously

Spend more time on “play” – it develops creativity – very much needed in the nonprofit sector

Center your policy, operations and decisions on your mission.

Have the Right State of Mind – From My Own 2012 Resolutions

I, of course, added my own two cents to the list which is simply to eliminate some of the “busyness” that takes so much time, and adds so little value to my life.   This busyness has been cluttering my brain and keeping me from dealing with issues in depth.  In 2012 I plan to fight back the sound bite life and give the “blue chips” more “in depth” attention.

I am moving into a new more spacious office space courtesy of my adult son moving into his own apartment. (Yeah!!!)  I am getting rid of stuff and getting more organized in my new space.  There is a place to continue to have candles lit when I am in a pensive mood or fresh flowers when I need to wake my spirits.  My husband, Hugh, did a scale floor plan with pieces for the furniture and in the big empty space in the middle he had a piece that said “Zach’s play area.”  Zach is my two and a half year old grandson and he does like to visit Grandma in her office.   (You can also read about my escapades as a grandmother at The Grandma Chronicles).  So I am looking forward to the type of year Anne talks about with “real meat on the value bones” and I plan to take Terrie’s advice and “add more play to my work.”

I hope you have found these ideas to be food for thought as you set your own goals for 2012.  Please share in the comments any additional thoughts or feedback on these ideas.

Marion

Program evaluations are “individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working¹.” What was your reaction to this definition?  Has the prospect of undertaking a “research study” ever deterred you for conducting a program evaluation?  Good news! Did you know that program evaluation is not the same as research and usually does not need to be as complicated?

In fact, evaluation is a process in which we all unconsciously engage to some degree or another on a daily, informal basis.  How do you choose a pair of boots? Unconsciously you might consider criteria such as looks, how well the boots fit, how comfortable they are, and how appropriate they are for their particular use (walking long distances, navigating icy driveways, etc.).

Though we use the same techniques in evaluation and research and though both methods are equally systematic and rigorous (“exhaustive, thorough and accurate”²), here are a few differences between evaluation and research:

Program Evaluation Focuses on a Program vs. a Population

Research aims to produce new knowledge within a field.  Ideally, researchers design studies to be able to generalize findings to the whole population–every single individual within the group being studied.  Evaluation only focuses on the particular program at hand. Evaluations may face added resource and time constraints.

Program Evaluation Improves vs. Proves

Daniel L. Stufflebeam, Ph.D., a noted evaluator, captured it succinctly:  “The purpose of evaluation is to improve, not prove³.” In other words, research strives to establish that a particular factor caused a particular effect.  For example, smoking causes lung cancer.  The requirements to establish causation are very high. The goal of evaluation, however, is to help improve a particular program.  In order to improve a program, program evaluations get down-to-earth.  They examine all the pieces required for successful program outcomes, including the practical inner workings of the program such as program activities.

Program Evaluation Determines Value vs. Being Value-free

Another prominent evaluator, Michael J. Scriven, Ph.D., notes that evaluation assigns value to a program while research seeks to be value-free⁴.  Researchers collect data, present results and then draw conclusions that expressly link to the empirical data.  Evaluators add extra steps.  They collect data, examine how the data lines up with previously-determined standards (also known as criteria or benchmarks) and determine the worth of the program.  So while evaluators also make conclusions that must faithfully reflect the empirical data, they take the extra steps of comparing the program data to performance benchmarks and judging the value of the program.  While this may seem to cast evaluators in the role of judge we must remember that evaluations determine the value of programs so they can help improve them.

Program Evaluations ask “Is it working?” vs. “Did it work”

Tom Chapel, MA, MBA, Chief Evaluation Officer at the Centers for Disease Control and Prevention (CDC) differentiates between evaluation and research on the basis of when they occur in relation to time:

Researchers must stand back and wait for the experiment to play out.  To use the analogy of cultivating tomato plants, researchers ask, “How many tomatoes did we grow?” Evaluation, on the other hand, is a process unfolding “in real time.”  In addition to determining numbers of tomatoes, evaluators also inquire about related areas like, “how much watering and weeding is taking place?”  “Are there nematodes on the plants?” If evaluators realize that activities are insufficient, staff are free to adjust accordingly.⁵

To summarize, evaluation: 1) focuses on programs vs. populations, 2) improves vs. proves, 3) determines value vs. stays value-free and 4) happens in real time.  In light of these 4 points, evaluations, when carried out properly, have great potential to be very relevant and useful for program-related decision-making.  How do you feel?

References:

1. U.S. Government Accountability Office.  (2005).  Performance Measurement and Evaluation.  Retrieved January 8, 2012 from http://www.gao.gov/special.pubs/gg98026.pdf
2. Definition of “rigorous.”  Retrieved January 8, 2012 from google.com
3. Stufflebeam, D.L.  (2007).  CIPP Evaluation Model Checklist.  Retrieved January 8, 2012 from http://www.wmich.edu/evalctr/archive_checklists/cippchecklist_mar07.pdf
4. Coffman, J.  (2003).  Ask the Expert: Michael Scriven on the Differences Between Evaluation and Social Science Research.  The Evaluation Exchange, 9(4).  Retrieved January 8, 2012 from http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/reflecting-on-the-past-and-future-of-evaluation/michael-scriven-on-the-differences-between-evaluation-and-social-science-research
5. Chapel, T.J.  (2011).  American Evaluation Association Coffee Break Webinar: 5 Hints to Make Your Logic Models Worth the Time and Effort.  Attended online on January 5, 2012

____________________________________________________________________________________

Priya Small has extensive experience in collaborative evaluation planning, instrument design, data collection, grant writing and facilitation.  Contact her at priyasusansmall@gmail.com.  Visit her website at http://www.priyasmall.wordpress.com.  See her profile at http://www.linkedin.com/in/priyasmall/