Western > Resources > Planning and Best Practices > Step 7 > Evaluation 5

V.  Evaluating Your Program Using the Logic Model

Now that you've selected some key evaluation questions it is time to make decisions about what information you need to answer the questions and how this information will be collected. The way that you collect evaluation information is called the evaluation method.

A. General Issues in Evaluation Methods

1. Types of Information

There are many different types of information that can be collected about the program. The following are some of the most common. We will be referring to these throughout the following sections. (Click on each for more information about that technique.)

• Written Surveys or Questionnaires: A written document containing questions that are answered by an individual. Survey questions can be open-ended (e.g., "What did you like about this program") or close-ended (e.g., "How much did you like this program (check one): ____ not at all ____ a little ____ somewhat _____ a lot"). Surveys and questionnaires can be used to evaluate many different aspects of a program.
  [more information on developing questionnaires]
• Interviews: Interviews involve one person asking questions of another person. Interviews can be done face-to-face or by telephone. Interviews, like surveys, can involve open-ended questions and close-ended questions. Interviews can be used in many different ways in an evaluation.
  [more information on interviews]
• Tests and Assessments: Tests and assessments are usually tools that have already been developed for particular purposes. They can provide information about different aspects of the evaluation but are most frequently used for assessing short- and long-term outcomes. Tests can include physical assessments, knowledge or achievement tests, psychological tests, and attitudinal assessments. Tests and assessments might be included as part of written surveys, questionnaires, or interviews. Through the use of standardized assessments on large segments of the population, test "norms" have been developed.
  [more information on tests and assessments]
• Observations: Observations of program activities are often used to understand the program context. Observational techniques can also be used to collect some forms of outcome data.
  [more information on observations]
• Focus Groups: Focus groups involve asking a series of predetermined questions to a group of people. People are encouraged to provide their opinions, but not necessarily to reach consensus. Focus groups can be used in a variety of ways in evaluation.
  [more information on focus groups]
• Case Study: Case studies involve collecting in-depth information about a few selected participants. Usually, case studies are used to provide "exemplars" of particular aspects of the program. Some evaluators also do case studies of programs.
  [more information on case studies]
• Document & Program Record Review: Existing program records and documents (such as meeting minutes, reports, brochures, etc.) can provide excellent information, particularly about program context and history, and about program implementation. Case files often contain a wealth of information about program participants that can be useful in the evaluation. Attendance records and staff logs often contain information about how many people have received services and how many services have been delivered.
  [more information on document & program record Reviews]
• Existing Databases and Archives: Many kinds of data are collected by public agencies, such as the Census Bureau, school districts, police departments, and other county, state and federal agencies. This data can be an important source of information for evaluation.
  [more information on existing databases and archives]

2. Quantitative and Qualitative Information

Quantitative data are reported in numerical form: test scores, numbers of people attending, rate of drop-out, or percentages. Quantitative data can be counted and measured. It is useful for describing concrete phenomenon and for statistical analysis of your results (e.g., calculating the percentage decrease in cigarette use among 8th graders). Surveys and questionnaires often include rating scales that can be used in quantitative analyses.

Qualitative data are reported in narrative form: written description of program activities, testimonials of program effects, comments about how a program was or wasn't helpful. Qualitative information can be used to describe how your project functions and what it may mean to the people involved. Qualitative data provide a context for the project and can be used to convey information about people's perceptions and reactions to a program.

Evaluation often requires measuring or assessing things that are hard to measure. In reviewing your evaluation questions, look for constructs that seem "vague" or "subjective." This doesn't mean that they aren't important to evaluate, but sometimes this can be quite difficult. Because of this difficulty, we recommend using a combination of qualitative and quantitative methods whenever possible. Most professional evaluators now acknowledge that both methods contribute important, complementary types of information to an evaluation.

3. Identifying Measurable Indicators

A challenge in any evaluation is identifying the information that best answers the question(s). Indicators express that which you wish to know. They are the measures or observable evidence that answer your questions.

Sometimes the needed information is obvious and straightforward (e.g., number and demographic profile of members, dollars generated, or description of services delivered). At other times the desired information is not specific enough to collect directly. For example, a short-term outcome question of interest might be: Did the program improve the quality of the relationship between parents and their children? "Quality of relationship" is a vague concept. You will need to define these terms before you can evaluate them:

• What does a positive parent-child relationship look like?
• How would you see it?
• How can you measure it?

It may also be important to understand how participants themselves define and understand key program outcomes. Consider this hypothetical example: Program staff defined and evaluated their outcome of a bilingual parent education program for Hispanic parents as knowledge gained (child development knowledge gained, proficiency in language). An evaluation showed little, if any, gains in knowledge.

Upon further probing, it was found that the participants were very satisfied with the program. For them, it had been very successful because at its conclusion they were able to share child-rearing tips with other parents and discuss the influence of American culture on their children. Parents felt that they had developed a support network of other parents that they could go to for advice and assistance in child-rearing. Staff-defined definitions of outcomes missed some important benefits as perceived by the participants.

It is usually important to engage different people with various backgrounds and cultural understandings in the identification of indicators. "Quality relationship," for example, has different meanings for different people in different places. Listening to multiple perspectives will produce indicators that are more complete and meaningful. Likewise, key stakeholders may have particular ideas about indicators that measure certain phenomenon. Your program will want to pay attention to these ideas if the results are to be used by these stakeholders. You may even wish to ask a few key individuals what evidence they think would answer the question.

4. Making Decisions about Evaluation Methods

The decisions about which methods to use will involve considering a number of factors. We make the following recommendations (adapted from the W.K. Kellogg Foundation):

1. Stay focused on the questions that you want to answer and select methods that are most appropriate to answering that question. Don't select your method before selecting your question!
2. Consider available resources. Usually, there is more than one way to collect information about any particular question. Some methods involve more time, money, and effort than others. Make realistic decisions about methods depending on your resources.
3. Be sensitive to the types of participants in your project and the kinds of information you are collecting. Some information may place participants at risk if it becomes public (e.g., any reporting of illegal activity). Other information may be sensitive to some people or groups; be sure and consider how respondents will feel if they are asked to provide particular information. You may also need to consider the particular skills of your respondents. If most don't read English, for instance, you will need to provide alternative ways of obtaining the information.
4. Credibility: How credible will your evaluation be as a result of the methods that you have chosen? Consider the following:
   • Is the instrument valid? That is, does it measure what it claims to measure? Do the questions make sense for your program?
   • Is the instrument reliable? That is, will it provide the same answers if administered at different times or places?
   • Are the methods suitable for the target group being studied?
   • Are the methods biased towards finding only positive results?
5. Does your program have the necessary expertise to collect the information?
6. Importance of the information: How important is a particular piece of information, in the context of your overall evaluation plan?
7. Who is going to be using the information? What type of information are they most likely to understand and consider credible? Will they be more receptive to statistics, to human stories, or to case descriptions, for example? In most cases, users find a mix of data/numbers and narratives—individual anecdotes and more generalizable results—most useful.

In your logic model, you should identify the information that you will need to answer each of your evaluation questions in a way that is clearly defined and measurable.

A Note about Measuring "Influential Factors"

As you think about your information needs, consider also the factors that are likely to influence the results. For example, in answering the question, "What did clientele gain from the collaborative service?" it is likely that what clients gain will be influenced by their demographic characteristics, skills and abilities, geographic location, or past experience with similar services. You will want to include such influential factors as data to collect if there is good reason to believe that such factors will influence your findings. However, avoid the temptation to collect data on everything. Think about what are likely to be the most important influential factors related to your evaluation question(s).

Previous | Outline | Next