Implementing the Evaluation

VII. Implementing the Evaluation

The evaluation implementation plan is a system for collecting evidence of program effectiveness. Writing an implementation plan will help you make sure you are collecting the information that you need in a timely and efficient way. The implementation plan describes the logistics of the data collection, and should address the following issues:

Who is responsible for developing the instruments?
When do the instruments need to be finalized?
Who will collect the data? Will the data be collected by program staff? Will volunteers be needed? Will you hire people to collect the data? Your data collectors may need training or other support to collect the information effectively. If staff are collecting additional information, be sure to support them by seeing the time spent on evaluation as a part of their workload.
What is the schedule for data collection?
- When will the information be available?
- When can the information be conveniently collected? When will it be least disruptive?
- Where will the information collection take place?
- When will data collection start and end?
Who will tabulate, organize, and analyze the information?
When will reports be generated?
Who will write the reports?

A. Who's Responsible for the Evaluation? Internal Vs. External Evaluations

Evaluations that are done by staff or volunteers who are also working for the program being evaluated are sometimes called internal evaluations. Evaluations done by hired consultants are sometimes called external evaluations. Another popular model for doing evaluations involves a combination of these two approaches, and is called participatory evaluation. Participatory evaluation usually, although not always, involves an external evaluator who works closely with program management, staff, volunteers, and participants to design the evaluation. The pros and cons to internal and external evaluations are summarized in the table below:

	Internal Evaluation	External Evaluation
Objectivity	Can be perceived as being less objective	Usually perceived as being more objective about the program because of lower level of direct connection with the program
Credibility	Can be seen as less credible due to lack of evaluation expertise	Usually perceived as more credible, provided sufficient time is taken to understand program functioning
Skills	Skilled and knowledgeable about program functioning	Skilled and knowledgeable about evaluation
Usefulness	May be more useful because of familiarity with the program	May be less useful because of lack of familiarity with programs
Success	May be more successful in getting support from other program staff	May be less successful in getting support from other program staff
Cost	Less expensive	More expensive

B. How to know if you need an Evaluation Consultant or Contractor

Although we believe that most programs can and should be engaged in their own internal program evaluation activities, there may be occasions in which the assistance of an evaluator is necessary. Here are some situations in which you might want to consider hiring an evaluation consultant or contractor:

The evaluation you want to do requires more time than program staff and volunteers can provide.
The program is "under fire" from outside critics, and the results of an internal evaluation might be subject to criticism.
You need statistical assistance in analyzing your data.
You need help in designing a questionnaire, survey, or other data collection instruments.
Your program is highly complex, and there are major disagreements about what the evaluation should focus on.

C. Finding and selecting a good consultant

Depending on your program's situation and needs, you will look for different kinds of skills in an evaluator. The table below (adapted from the W. K. Kellogg Foundation) helps to identify the kinds of skills you might need to look for, depending on your program's situation:

Program Situation	Challenge	Evaluator Skills to Look For
Controversy over the program and how to evaluate it	Facilitating different points of view	Conflict resolutions skills Team-building skills Neutrality & Objectivity
Highly visible program	Dealing with publicity, reporting findings in a political environment	Public presentation skills Experience with media and politicians Credible reputation and experience
Flexible, highly dynamic program	Adapting to rapid changes in context, issues, focus	Tolerance for ambiguity Flexibility Skills in qualitative methods Quick learner
Collaborative program (team effort)	Managing people	Team-building skills Ability to focus and direct progress Experience in participatory evaluation

Previous | Outline


Privacy Policy \| Site Disclaimer \| Site Accessibility
SAMHSA \| NCADI \| National Mental Health Information Center \| USA.gov Page last updated: 11/13/2008