Now that you understand the basics of evaluation design, and the value of conducting a well-planned—and well-executed—assessment, you can begin thinking about your role in the overall evaluation process. There are two primary approaches to conducting an evaluation: traditional and participatory. The approach you select will directly influence the type of evaluator you choose to work with.

Traditional Evaluation
In traditional evaluation, the evaluator is responsible for all decisions about how to conduct the evaluation. He or she is the “expert,” working at an arm’s length distance from the program. Evaluator contact with program staff is minimal, and staff is typically excluded from evaluation tasks. Staff may help arrange for administration of a survey or other data collection tasks, but, by and large, the evaluator alone does the work. For many years, traditional evaluation—seen as the most efficient and objective approach—was the only model used.

A relatively new but increasingly valued approach involves a partnership between the evaluator and the people developing and delivering program activities. This approach is known as collaborative or participatory evaluation.

Participatory Evaluation
One of the negative connotations often associated with evaluation is that it is something done to people. One is evaluated. Participatory evaluation, in contrast, is a process controlled by the people in the program or community. It is something they undertake as a formal, reflective process for their own development and empowerment. (Patton, 1990, p.129)

Participatory evaluation relies on an evaluation team composed of one or more individuals trained in evaluation, program staff, and other stakeholders (e.g., involved community members, police officers, local politicians). The team should also include representatives of all relevant ethnic and racial groups. Members of the team are involved in all phases of program development, from the conceptualization of problems to the evaluation and interpretation of findings.

Participatory evaluation builds on the strengths and values the contributions of everyone involved. It can help you produce a better evaluation for the following reasons:

• It promotes shared expectations. When an evaluation plan is developed collaboratively, there is less potential for misunderstanding and an increased likelihood that all involved will be on the same page.
  
  
• It capitalizes on staff investment. In general, program staff and stakeholders have a greater personal investment in the evaluation than the evaluator—particularly if their job depends on the outcome of the evaluation. Their ongoing participation helps ensure that the program will be assessed carefully and treated fairly.
  
  
• It increases the program’s evaluation capacity. Staff members who participate in evaluation activities will have a better understanding of good research, and be in a better position to conduct their own research and review the research of others. In addition, the more they know about the complexities of survey design and data analysis, the more forgiving they will be of the associated costs.
  
  
• It increases the likelihood that research questions and data collection methods will be appropriate and relevant to the target populations. Program staff and stakeholders may be more knowledgeable than an evaluator about the needs, culture, and circumstances of the target population. This knowledge will improve the overall quality of the evaluation design, including data analysis. Failure to take cultural dimensions into reasonable consideration can lead to misinterpretations of the evaluation data.

• It increases the chance that results will be used. When staff members are integrally involved in the evaluation process, they are more likely to understand, accept, and apply the findings of the evaluation. Rather than receiving (and resisting) an outside evaluation report, the process of participating in an evaluation gives ownership of the information to those most involved in carrying out the work of the organization.
  
  
• It helps to legitimize the evaluation, particularly when you include on your evaluation committee primary users with the organizational authority and power to act on data once it is generated.
  
  
• It is more flexible and less costly than traditional models. With more people involved, the balance of responsibilities can shift as needed throughout the duration of the project. Furthermore, with program staff on board to help, fewer tasks fall to the more costly evaluator.

Yet participatory evaluation can also raise problems. Below are some possible challenges to this model and strategies that address them:

• The potential for bias may be increased. To reduce this risk, think critically about who should and should not carry out specific evaluation tasks. For example, a school administrator should not interview teachers about how well they implemented a prevention program. The teachers are more likely to be comfortable and candid discussing their implementation of program activities with an evaluator. Remember that it isn’t really the players that keep bias from being a problem—it’s how the study is conducted. If the design is solid, then the results can be relied on.
  
  
• Participants may disagree about the distribution of labor. It is sometimes difficult to establish a balance of responsibilities that works for everyone. To minimize this problem, try to be clear from the outset who will be doing what.
  
  
• Program staff may not want to take on additional responsibilities. They may feel that they have too much to do already and that evaluation is not part of their job. To increase buy-in, explain the importance of the evaluation and how staff involvement will contribute to its success.
  
  
• Collaboration takes time. Yet, remember, the time spent meeting, consulting, and sharing points of view will result in a more relevant and useful evaluation. Also keep in mind that increased meeting costs can be more than offset by the savings that result from having staff carry out evaluation tasks.
  
  
• It may be more difficult to locate an appropriate evaluator. Not all evaluators are committed to or even interested in collaborating with program staff and stakeholders. They may not be accustomed to sharing power or feel comfortable working cooperatively with individuals who differ from them in terms of education, class, culture, race, or ethnicity. If you would like to undertake a participatory evaluation of your program, make sure to discuss this approach with candidates during the screening process. The topic of locating and hiring an appropriate evaluator will be covered in greater detail tomorrow.

The type of approach you select will depend on a variety of factors, including cost, staff abilities, and staff willingness to participate. Each approach embodies different values and results in different relationships, processes, and outcomes. In the end, you must work with your planning team and other key partners to determine the best approach for the evaluation of your program.

Please proceed to Activity 2: Selecting an Approach.

References

Center for Substance Abuse Prevention(1996). CSAP cultural competence series 6: Advanced methodological issues in culturally competent evaluation for substance abuse prevention (DHHS Publication No. (SMA)96-3110). Rockville, MD: Center for Substance Abuse Prevention, U.S. Department of Health and Human Services.

Harding, W. (2000). Locating, hiring, and managing an evaluator. Newton, MA: CSAP’s Northeast Center for the Application of Prevention Technologies.

Jackson, E. T. (1998). Introduction. In E. T. Jackson and Y. Kassam (Eds.), Knowledge shared: Participatory evaluation in development cooperation (pp. 1-20). West Hartford/Ottawa: Kumarian/IDRC.

W. K. Kellogg Foundation (1998). Evaluation handbook. Battle Creek, MI: Author. Available online at http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf.

Patton, M. (1990). Qualitative evaluation and research methods (2nd ed.), Thousand Oaks, CA: Sage.