IV. How to Plan Your Evaluation
A. General Considerations
The first step to any good evaluation is careful planning. You will need to answer a number of questions in order to make good decisions about how to conduct the evaluation. First, however, we make some general recommendations:
In order to plan your evaluation, you will need to answer the following questions:
1. What are you going to evaluate?
The first question that you need to answer is what you want to evaluate. This may not be easy depending upon how clearly defined your program is. For example, you might wish to evaluate a comprehensive prevention program that involves student education, community awareness, and information dissemination. Think about the program's purpose and content. Do you want to examine the whole program or just a particular component of it? Do you want to examine all activities or just selected ones? Often, it isn't necessary to evaluate all components of a comprehensive program.
Some factors to consider:
- Which program elements are most likely to have the strongest effects on participants? For example, a one-shot motivational speaker is not likely to have significant long-lasting influence on youth attitudes. However, a year-long sequence of classes is likely to have a strong influence.
- Which program elements require the greatest proportion of your resources? Aspects of a program that consume high proportions of resources might be more important to evaluate.
- Which program components have a strong research basis that suggests they are effective? If strong research evidence shows that a program is effective if implemented properly, you may need to evaluate only implementation, and focus less on outcomes.
- Feasibility of evaluation. Some prevention activities are likely to be extremely difficult to evaluate, such as media campaigns that reach thousands of homes, or awareness marches. Your evaluation resources might be better used to evaluate other aspects of programming for which evaluation will not be as difficult or costly.
Your logic model will be helpful in guiding you towards the most important program elements to evaluate!
2. What do you (and others) want to know about the program?
What questions do you want to answer? The cornerstone of any good evaluation are your evaluation questions. That is, what are you trying to find out by doing the evaluation? In order to formulate your evaluation questions, we recommend that you first consider the following issues:
a. What is the purpose of the evaluation
b. Who wants to know what?
c. When do you need the information?
d. What resources do you have available for evaluation?
There are at least three major purposes for evaluation. We refer to these as:
Monitoring: Monitoring is done for the purpose of documenting program implementation. Monitoring is also sometimes called "process" evaluation, and is usually done for the sole purpose of documenting whether a program is being implemented as planned.
Formative Evaluation: Formative evaluation is done for the purpose of continuous program improvement. Although sometimes the term "formative" is used interchangeably with "process," this is not correct. A formative evaluation typically involves collecting a combination of information about program activities, outputs, barriers to implementation, client satisfaction, and outcomes. What distinguishes a formative evaluation is that it implies that the information is used for ongoing continuous program improvement.
Summative Evaluation: Summative evaluation is done for the purpose of learning whether the program "works." Summative evaluations, sometimes also called "impact" evaluations, are focused on demonstrated program outcomes and impacts only (although monitoring is often done in tandem with summative evaluation).
A cautionary note about summative evaluations: We don't recommend doing summative evaluation only. This is what is sometimes called a "black-box" evaluation. Evaluations that focus only on outcomes or impacts without information about how the program was implemented, how clients experienced the program, etc., may be of limited value, especially if your outcomes/impacts aren't as positive as you had hoped. A summative evaluation without a monitoring or a formative component won't provide information about why the program worked or didn't work.
b. Defining the users of the evaluation: Who is going to be using the information?
Have you ever completed an evaluation and wondered, "Now what should I do with it?" Or tallied the results but never really used the findings? If you want to collect relevant data and make the best use of limited resources, you must think about how you'll use the evaluation right from the start.
Sometimes, we conduct an evaluation only for internal program use. Usually, however, there are others who have requested or could use the resulting information. Any of the stakeholder groups listed below might be interested in the evaluation results of a prevention program:
- People affected in some way by the program (either directly or indirectly), such as program participants, nonparticipants, critics
- County board members, elected officials
- Community leaders and activists
- Volunteers, collaborators, supporters
- Program managers and staff
- Media
- Program funders
- Schools and other community providers
It is important to identify potential users of the information. Find out what they want to know, and how they will use the information (see table below). If you don't know, ask. This will help you to clarify the purpose(s) of the evaluation, build commitment for it, and fine-tune the particular questions the evaluation will address.
The chart below demonstrates the range of users, questions, and uses that might arise. It also demonstrates the need to prioritize evaluation concerns.
User of evaluation
Questions of interest
Uses of information
Public officials
What is the program doing?
What difference is it making?
Is it reaching the target population?
What do clientele think about the program?Decisions about commitment and support
Knowledge about usefulness of program approachCommunity residents
Is the program suited to my community's needs?
What is the program doing?Decisions about whether to participate and/or contribute support
Public officials
Did the program help me?
What could improve the program for others?Decisions about continuing with program, whether to participate in similar initiatives
Program Directors & Managers
Are we reaching our target population?
Are people satisfied with our programs?
Is the program being run efficiently?
How can we improve the program?Programming decisions, day-to-day operations
Funders
Is what was promised being achieved?
Is it working?Accountability
c. Defining the evaluation questions
Once you've identified the purpose of the evaluation and the key questions of various stakeholders, you can use the logic model to identify key questions about each step in the model. You can develop questions about each column in your logic model, although in all likelihood you won't ever evaluate all the possible questions you identify. Most programs find it easier to generate a large list of possible questions and then focus the evaluation on a few key questions. We recommend generating this large list of questions because it will help ensure the really important questions are included in the evaluation.
As you think about the questions that your evaluation will answer, it may be necessary to break a larger question into its component parts. This will help you fully answer the broader question and begin to identify the information you need to collect. Consider the following examples:
Main question: Who benefits from the program ?
Sub-questions: Who actually participates in the program? At what level of involvement? Who else gains from the program? What do they gain? How do program participants compare to the county population in general? Who may be negatively affected? How?Main question: Is the program duplicating other efforts ?
Sub-questions: Of what does the program consist? What other similar programs exist—of what do they consist? How are aspects of these programs alike? Dissimilar? Complementary? What is our particular expertise / niche ?Main question: Did people learn the importance of X?
Sub-questions: Did people know anything about X before attending the program? Was the environment conducive to learning? Do any other programs or agencies promote the importance of X?Sample Evaluation Questions From Hypothetical Logic Models:
- Evaluation questions about program activities
- Evaluation questions about focus groups
- Evaluation questions about theories of change
- Evaluation questions about short-term outcomes
- Evaluation questions about impacts
3. Focusing the Evaluation: What questions are you going to answer now?
No program evaluation can ever address all of the questions that different stakeholders might want to know about a particular program. The next step, therefore, is to prioritize your questions.
Try to distinguish between what you need to know and what might be merely nice to know. Focus on the key questions that are most important. As needed, bring stakeholders together and negotiate a practical list. Above all, keep the evaluation manageable. It is better to stay focused and answer a few questions well. There are several important things to consider when deciding which questions you would like to answer, in addition to the considerations described above (the evaluation purpose and stakeholder questions).
a. Timing and Program Development
One common mistake in evaluation is to jump immediately into evaluating outcomes and impacts, without collecting basic information about program implementation. Even though questions about outcomes and impacts may be extremely important to you, we do not recommend evaluating them in the absence of good information about program implementation. Therefore, we recommend that every evaluation collect at least some monitoring information about the number, type, and intensity of services provided.
Jacobs (1988) recommends that evaluation questions fit the "developmental" stage of the program. New programs often take some time to get off the ground. Evaluations of new programs in their early startup stages, therefore, should focus more strongly on program implementation and formative evaluation questions focused on program startup.
Formative evaluation—which focuses on providing ongoing, timely feedback to programs about both implementation and outcomes—is often more appropriate for new programs because the information can help identify barriers to implementation and fine-tune program activities to meet the needs of participants.
Summative evaluation is more appropriate for testing the effectiveness of programs that have been fully operational for some period of time so that any problems with implementation have been addressed.
The evaluation effort should fit the programming effort. In some cases, you may only be interested in finding out how people responded to alternative activities, or how satisfied they were with a particular event. In other cases, you may want to document behavioral changes or impacts which require a more comprehensive evaluation and level of effort. The point is to tailor your evaluation to fit the program. Don't expect to measure impact from a single workshop or behavioral changes from a limited media effort.
Remember, all components of every program do not need to be formally evaluated. Formal evaluations require time, money, and resources. Sometimes programs lack sufficient substance to warrant a formal evaluation, or it may be too costly to collect the evidence needed to demonstrate impact. It may be possible that no one is interested in the findings.
You also need to consider the people participating in your programs. People get tired of filling out end-of-session forms or answering surveys. Be selective and considerate. Think about what is needed and what will be used.
c. Pragmatic Considerations: Time and Money
When is the evaluation information needed? Different kinds of evaluations require that information be available at different points in time: Monitoring: Monitoring can be used for two main purposes: reporting to funders (e.g., accountability), or for continuous program improvement (e.g., monitoring the number of clients served and using this information for allocating program resources). If the information is required for continuous program improvement, the information must be available more quickly.
Formative Evaluation: If the purpose of your evaluation is continuous program improvement, you will need to collect information that is reported back to the program in a timely manner. Whether the information is about program implementation or outcomes, timeliness is the key. A good formative evaluation will provide short, regular, updates to the key program stakeholders about how the program is doing, often in the form of brief reports, email updates, and informal reports. Formative evaluation assumes that you can't wait until a program is over before information is available. Rather, information should be available during program implementation.
Summative Evaluation: If the purpose of your evaluation is to document whether the program achieved its outcomes or not, the information will typically not be available until after the program is completed.
Deadlines: When the information is needed and what you can manage to do within the timeline will influence the scope of your evaluation. You might decide to save some questions or concerns for another study; or to discard others as unnecessary or inconsequential. Try to develop a realistic timeline for completing the evaluation. Keep the components of your plan manageable so they can be handled well within those time limits.
Usable moments: You may not need evaluation information to meet a specific deadline, but there might be times when having such information would serve a valuable purpose. A few examples of such "usable moments" include important committee meetings, testimony in front of offenders, pre-budget hearings, etc.
When making decisions about which evaluation questions to focus on, you should consider when you will need the information. What resources do you have for evaluation (time, money, people)? The resources you have available may influence your evaluation plan more than any other single factor. Even if you expect to integrate evaluation into the program or if the evaluation will be conducted by volunteers or the participants themselves, you will need to allocate time for planning. Balancing your expectations (and those of others) with what is realistic and manageable is a challenge. You'll need to consider:
- Time. Whose time and how much of it is available to work on evaluation? What priority will evaluation have in your overall workload? Involving volunteers or participants is a way to spread the workload, but it may require time for preparation or training.
- Money. Some activities require financing. For example, what dollar resources are available to print questionnaires, pay for postage, reimburse participants, analyze the data?
- Expertise. Sometimes we need outside expertise to help with certain parts of an evaluation. Constructing the instrument or analyzing the data may require such help. Or, there may be others with a lot of experience and knowledge related to the program from whom you could learn. Sometimes the involvement of an "outsider" increases the evaluation's credibility. We discuss this issue more completely below. When deciding which evaluation questions to answer, you need to consider the resources available for actually conducting the evaluation. Make sure that the questions you want to answer are reasonable, given the available resources.
A note about resources: As professional evaluators, the question we are most often asked is "how much does evaluation cost"? This is an extremely difficult question. The best answer we can give is "it depends"— on the questions you have, the kind of program you're evaluating, the confidence with which you need to know your answers, and the methods that you choose to answer the questions. Evaluations can be done by existing staff using existing data; the cost of this evaluation is primarily the cost of some staff time to put the information into a useable form. Evaluations can be done by external consultants using labor-intensive methods over long periods of time. In some national demonstration projects, the cost of the evaluation is equal to the cost of the program itself.
Typically, agencies do not have large amounts of money available for evaluation. In these cases, the cost to the agency is in staff time. We urge the people planning the evaluation to remember that if the evaluation is to be done well, it will require staff time. Staff assigned to work on the evaluation should accordingly be given a reduction in their regular workload. Evaluations which are simply "added on" to an already full workload are likely to fail.
|
|
