Free Management Library's
Online Nonprofit Organization Development Program Module #11:

Designing Your Program Evaluation Plans
(including outcomes-based evaluations)

© Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC.

Much of this program is based on materials adapted from the
Nonprofit Capacity Building Toolkit(SM)

(Much of the information in this module is adapted fromField Guide to Nonprofit Program Design, Marketing and Evaluation.)

This learning module is in the nonprofit organization development program. However, this module can also be used by anyone as a self-study exercise to learn more about nonprofit program evaluation.)

INTRODUCTION

The concept of program evaluation can include a wide variety of methods to evaluate many aspects of programs in nonprofit or for-profit organizations. There are numerous books and other materials that provide in-depth analysis of evaluations, their designs, methods, combination of methods and techniques of analysis. However, personnel do not have to be experts in these topics to carry out a useful program evaluation. The "20-80" rule applies here, that 20% of effort generates 80% of the needed results. It's better to do what might turn out to be an average effort at evaluation than to do no evaluation at all. (Besides, if you resort to bringing in an evaluation consultant, you should be a smart consumer. Far too many program evaluations generate information that is either impractical or irrelevant -- if the information is understood at all.) This learning module orients personnel to the nature of program evaluation and how it can be carried out in a realistic and practical fashion.

NOTE ABOUT BOARD COMMITTEES: Consider using the Board Programs and Marketing Committee to review and help guide implementation the information in this learning module. Major activities and goals from this learning module could be incorporated in that Committee's Committee Work Plan.

---

MATERIALS TO REVIEW

• The following materials will help you address each of the topics and learning activities in this module.

Basic Guide to Program Evaluation (about evaluation, in general) - - - including the sections:
- - - Program Evaluation: carefully getting information to make decisions about programs
- - - Where Program Evaluation is Helpful
- - - Basic Ingredients (you need an organization and program(s))
- - - Planning Program Evaluation:(what do you want to learn about, what info is needed)
- - - Major Types of Program Evaluation (evaluating program processes, goals, outcomes,)
- - - Overview of Methods to Collect Information (questionnaires, interviews, focus groups.)
- - - Selecting Which Methods to Use (to get needed info from audiences)
- - - Analyzing and Interpreting Information (to make conclusions and recommendations)
- - - Reporting Evaluation Results
- - - Who Should Carry Out the Evaluation?
- - - Contents of an Evaluation Plan
- - - Pitfalls to Avoid

- - - including the sections:
- - - Reasons for Priority on Outcomes-Based Evaluation
- - - Basic Principles for Small Nonprofits to Remember Before Starting Outcomes Planning
- - - What is Outcomes-Based Evaluation?
- - - Myths to Get Out of the Way Before You Start Your Outcomes Planning
- - - Planning Any Type of Evaluation Includes Answers to These Very Basic Questions
- - - Planning Your Outcomes Evaluation -- Step 1: Getting Ready
- - - Planning Your Outcomes Evaluation -- Step 2: Choosing Outcomes
- - - Planning Your Outcomes Evaluation -- Step 3: Selecting Indicators
- - - Planning Your Outcomes Evaluation -- Step 4: Planning Data/Info Collection
- - - Planning Your Outcomes Evaluation -- Step 5: Piloting/Testing
- - - Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results

---

SUGGESTED TOPICS FOR REFLECTION AND DISCUSSION

• Learners are strongly encouraged to discuss the following questions with peers, board members, management and staff, as appropriate.

Program Evaluation in General

1. What is "program evaluation"? Name at least three of its benefits -- or three situations when it is helpful? (See Program Evaluation and Where Program Evaluation is Helpful.)

2. What are the two key ingredients you need to have before you can conduct program evaluation? (HINT: The answer may be simpler than you think, but it's surprising how many nonprofit do not have both of these ingredients before they set out to conduct program evaluation!) (See Basic Ingredients.)

3. What determines how you will conduct your program evaluation? (HINT: Your program evaluation plans depend on what _____ you need to collect in order to make major_____.) (See Planning Program Evaluation.)

4. Name at least four of the seven key considerations when designing program evaluations. (See Planning Program Evaluation.)

5. What are three major types of evaluations typically done when carrying out nonprofit program evaluations? (There are many types of evaluations. This question asks about major types of evaluation used in program evaluation in nonprofits.) (See Major Types of Program Evaluation.)

6. Name at least four of the fix major methods of data collection as listed in the information about program evaluation. (See Overview of Methods to Collect Information .)

7. Name at least five of the nine questions that should be considered when selecting the evaluation method (when conducting program evaluation). (See Selecting Which Methods to Use.)

8. What are the four levels of information that can be gathered from people during a program evaluation? (See Selecting Which Methods to Use .)

9. Name at least three of the five pitfalls of program evaluation planning. (See Pitfalls to Avoid.)

Outcomes-Based Evaluation

1. What are the major reasons that outcomes-based evaluation is a top priority among nonprofit funders and service providers today. (See Reasons for Priority on Outcomes-Based Evaluation.)

2. What are program inputs? Activities? Outputs? Outcomes? Outcome targets? Indicators? (See What is Outcomes-Based Evaluation? )

3. What is the difference between outputs and outcomes? Outputs and indicators? Indicators and outcome targets? (See What is Outcomes-Based Evaluation? )

4. Name at least three of the six myths listed in the suggested materials for review. (See Myths to Get Out of the Way Before You Start Your Outcomes Planning)

5. Name at least five of the typical questions answered in almost any type of evaluation. (See Planning Any Type of Evaluation Includes Answers to These Very Basic Questions.)

6. Name the guidelines for picking which program to evaluate. (See Planning Your Outcomes Evaluation -- Step 1: Getting Ready)

7. What is a logic model? (See Planning Your Outcomes Evaluation -- Step 2: Choosing Outcomes.)

8. What should you look for in short-term outcomes? Intermediate outcomes? Long-term outcomes? (See Planning Your Outcomes Evaluation -- Step 2: Choosing Outcomes.)

9. What is a questionnaire? Interview? Case study? (See Planning Your Outcomes Evaluation -- Step 4: Planning Data/Info Collection.)

10. What are some practical considerations when selecting which data collection method to use? (See Planning Your Outcomes Evaluation -- Step 4: Planning Data/Info Collection.)

11. What should you always do with the data that you've collected before you analyze it? (See Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results.)

12. What ultimately determines the level and scope of the evaluation report? (See Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results.)

13. Name at least five of the sections of a typical evaluation report. (See Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results.)

---

ACTIVITIES TO BUILD SYSTEMS AND PRACTICES

• Learners are strongly encouraged to complete the following activities, and share and discuss results with peers, board members, management and staff, as appropriate.
• As you proceed through the following activities, be sure to note any incomplete actions in the Action Item Planning List.

Writing Your Overall Program Evaluation Plan

Complete the following checklist about each of the program that you would like to evaluate.

Purpose of Evaluation?
What do you want to be able to decide as a result of the evaluation? For example:
__ Understand, verify or increase impact of products or services on customers/clients (eg, outcomes evaluation)
__ Improve delivery mechanisms to be more efficient and less costly (eg, process evaluation)
__ Verify that we're doing what we think we're doing (eg, process evaluation)
__ Clarify program goals, processes and outcomes for management planning
__ Public relations
__ Program comparisons, eg., to decide which should be retained
__ Fully examine and describe effective programs for duplication elsewhere
__ Other reason(s)

Audience(s) for the Evaluation?
Who are the audiences for the information from the evaluation, for example:
__ Clients/customers
__ Funders/Investors
__ Board members
__ Management
__ Staff/employees
__ Other(s) _________________________________________________

What Kinds of Information Are Needed?
What kinds of information are needed to make the decision you need to make and/or enlighten your intended audiences, for example, information to understand:
__ The process of the product or service delivery (its inputs, activities and outputs)
__ The customers/clients who experience the product or service
__ Strengths and weaknesses of the product or service
__ Benefits to customers/clients (outcomes)
__ How the product or service failed and why, etc.
__ Other type(s) of information?

Type of Evaluation?
Based on the purpose of the evaluation and the kinds of information needed, what types of evaluation is being planned?
__ Goal-based?
__ Process-based?
__ Outcomes-based?
__ Other(s)? ___________________________________________________

Where Should Information Be Collected From?
__ Staff/employees
__ Clients/customers
__ Program documentation
__ Funders/Investors
__ Other(s) ________________________________________________

How Can Information Be Collected in Reasonable and Realistic Fashion?
__ Questionnaires
__ Interviews
__ Documentation
__ Observing clients/customers
__ Observing staff/employees
__ Conducting focus groups among_____________________________________
__ Other(s)

When is the Information Needed?
________________________________________________________________

________________________________________________________________

What Resources Are Available to Collect the Information?

________________________________________________________________

________________________________________________________________

Writing Your Outcomes-Based Evaluation Plan

Complete each of the steps specified in the following sections. You will be asked to include some of your answers in the Framework for a Basic Outcomes-Based Evaluation Plan, which will culminate in your developing your basic outcomes-based evaluation plan.

1. Planning Your Outcomes Evaluation -- Step 1: Getting Ready

2. Planning Your Outcomes Evaluation -- Step 2: Choosing Outcomes

3. Planning Your Outcomes Evaluation -- Step 3: Selecting Indicators

4. Planning Your Outcomes Evaluation -- Step 4: Planning Data/Info Collection

5. Planning Your Outcomes Evaluation -- Step 5: Piloting/Testing

6. Planning Your Outcomes Evaluation -- Step 6: Analyzing/Reporting Results

---

ASSESSMENT(S)

1. In the next module in the program, you will conduct a complete "fitness test" of your nonprofit. See Complete Fitness Test of Your Nonprofit.

---

REMINDERS FOR THOSE IN THE ON-LINE DEVELOPMENT PROGRAM

1. Are you exchanging feedback with others about what you're learning in this program?

2. Are you sticking to your study schedule for this program?

3. Are you practicing your basic skills in management and leadership, including in problem solving and decision making, planning and meeting management?

4. Are you communicating throughout your organization by using your skills in internal communications?

5. Are you managing yourself? How many hours a week are you working? Are you noticing any signs of stress? If so, what are you doing about it?

6. One of the ways you might be able to tell if you're stressed out and/or losing perspective might be whether you're tracking details or not. Are you using the action item list referenced above?

---

TRACKING OPEN ACTION ITEMS

1. One of the first indicators that an organization is struggling is that open action items are not tracked and reviewed. (Open action items are required actions that have not yet been completed.) Instead, organization members only see and react to the latest "fires in the workplace". Whether open action items are critical to address now or not, they should not entirely be forgotten. Therefore, update and regularly review a list of open action items that includes listing each open action item, who is responsible to complete it, when it should be completed and any associated comments. When updating the list, consider action items as identified during discussions, learning activities and assessments in this module. Share and regularly review this action item list with the appropriate board, management and staff members in your nonprofit. You can use the following Action Item Planning List.

2. What remaining questions or comments do you have that you want to pose? Pose them in ODNet-Nonprofits, a free online discussion group..

---

(Learners in the nonprofit organization development program can return to the nonprofit organization development program.)

---

---

For the Category of Evaluations (Many Kinds):

Related Library Topics

Recommended Books

General Information (Applying to Many Types Evaluation)

The following books are recommended because of their highly practical nature and often because they include a wide range of information about this Library topic. To get more information about each book, just hover your cursor over the image of the book. A "bubble" of information will be displayed. You can click on the title of the book in that bubble to get more information, too.

---

Also see

For evaluating employees, Supervision -- Recommended Books

For evaluating nonprofit programs, Program Management -- Recommended Books

---