Evaluation is one area where it often makes sense to bring in outside expertise. It will save you considerable time and effort, and will increase your probability of success significantly. In fact, the assistance of an evaluator attuned to and practiced in the art and science of theory-based evaluation is often the key to getting evaluation results that are useful and credible. The purpose of today’s presentation is to provide you with the information and tools you need to locate and select an evaluator whose skills match your programmatic needs—assuming, of course, that you don’t already have an evaluator on staff.

Starting the Search
The first step in selecting an evaluator is to decide whether you want to work with an in-house evaluator (someone employed by your organization) or an external consultant. Many of you may not be in a position to make this decision—either you lack in-house evaluation capacity or the option to look elsewhere. For those with the flexibility to choose, the following chart, which compares internal and external evaluators according to a variety of criteria, can help you do so.

It’s important to note that some of the information in this table assumes that (1) the internal evaluator is not an evaluation expert, and (2) the external evaluator is taking a traditional, rather than participatory, approach. You can increase the likelihood that your evaluation will be successful by working with either a highly skilled internal evaluator or an external evaluator committed to a truly collaborative approach. You might also decide to combine both options by having an internal staff person conduct the evaluation and an external consultant assist with the technical aspects and help gather specialized information. With this combination, the evaluation can provide an external viewpoint without losing the benefit of the internal evaluator’s first-hand knowledge of the project.

Locating an Evaluator
If you’ve never before looked for an evaluator, the prospect of doing so may feel overwhelming. In fact, many people, eager to end the search process as quickly as possible, hire the first evaluator they contact. Resist this temptation! Not all evaluators have the same set of skills you may need. There are many ways and places to locate qualified candidates; the more thorough your search, the more likely that you will find an evaluator who is a good match for your project.

Figuring Out What You Want—and What You Can Live With
You’ve begun collecting names, but now what? Before scheduling your first interview, sit down with your planning team and talk through what you’re looking for in an evaluator. Should the evaluator be flexible? Cheap? Articulate? And how important is cost relative to flexibility? Are there certain traits that are particularly important, given your project? For example, if you are working with a largely Latino population, is it important that your evaluator speak Spanish? Remember, no new hire—for any job—can do and be everything to everyone. Identifying and prioritizing your selection criteria ahead of time will prevent you from interviewing candidates who aren’t right for the job or looking for something that is impossible to find.

Assessing Your Candidates

Whatever your past experiences, hiring an evaluator should be a competitive process. Take the time to recruit candidates that meet your criteria, then carefully assess their qualifications. The conversations you have up-front, during the interview process, will lay the groundwork for how you go on to work together.

Key items that you should make a point of discussing with potential candidates include the following:

• Evaluation philosophy. Consider the evaluation philosophy or approach that you find most comfortable and appealing. If you select the participatory approach, it is important that your evaluator will be comfortable with this level of collaboration. Ask candidates if they have used this approach before and about their experiences doing so.
  
  
• Education and experience. If you cannot identify a candidate with formal training in program evaluation, look for individuals with graduate-level training in social science research methods. They should also have professional experience in the areas of evaluation design, data collection, and statistical analysis. Ideally, candidates will have additional experience that is relevant to your specific program. Ask candidates whether or not they have evaluated similar programs with similar target populations. If they have, then they probably have knowledge and resources (e.g., appropriate data collection instruments) that can save you both time and money. To get a clear sense of their work, ask to see evaluation reports that they have prepared.
  

Communication skills. Evaluators must be able to communicate effectively with a broad range of audiences (e.g., parents, a school board, other evaluators, the media). They should avoid jargon; someone who cannot clearly explain evaluation concepts is not a good candidate. To gather accurate information, an evaluator needs to be able to connect comfortably with program staff and participants. He should be personable and engaging, as well as capable of making evaluation results both compelling and accessible. You also want to make sure that the evaluator you hire has strong writing skills. Check this by having candidates bring samples of past reports they have written (including the executive summaries), articles, and presentations that they have developed to share their findings. Ideally, you should also ask candidates to prepare a written proposal for your evaluation—though you might want to reserve this request for your pool of finalists. To get a good proposal, provide candidates with clear information about your program’s goals, activities, and audience.

Is it ever okay to hire an evaluator without relevant experience? Sure! But if you do, be sure you’re not going to be paying to educate that person. If evaluators want to work in a new area, the cost of learning the ropes should be theirs. How about someone with little experience? That may have to be an option, particularly if cost is an issue—but you are taking a risk. Have a straightforward conversation with candidates about their lack of experience. You could suggest that they charge less because you are giving them the chance to branch out into a new field. You might also hire a more experienced evaluator to have a consultation meeting with you and the less experienced evaluator to create an evaluation design and overall plan of action.

• Cultural sensitivity. An evaluator needs to respect the cultures of the communities with which he or she works. Mutual respect and some understanding and acceptance of how others see the world is crucial. Genuine sensitivity to the culture and community will help increase the comfort level of program staff, participants, and other stakeholders. It will also ensure that data collection tools are appropriate and relevant, thus increasing the accuracy of the findings.
  

• Budget and cost. Ask for a detailed budget that distinguishes between direct costs, such as labor, and indirect costs, such as overhead. Overhead rates vary widely. It is not unusual to see overhead costs of 100 percent or more, meaning that for every dollar that goes toward conducting the study, another dollar goes toward running the organization responsible for the study. Sometimes you can get an organization to reduce its indirect costs. This saves you money without compromising the quality of your study. (We will be discussing additional ways to reduce evaluation costs on Day 4.)

• Time and access. Make sure candidates have the time to complete the necessary work. Ask them about their current work commitments and how much time they will be able to devote to your project. Compare their responses to your estimate of the time needed to do the work. Make sure to factor in frequent site visits and regular meetings. The more contact your evaluator has with your program, the better he or she will understand how it works and the more opportunities he or she will have to monitor data collection activities. Regular meetings also let you monitor the evaluator’s performance. If the evaluator is not local, travel expenses may increase the cost of your evaluation.
  
  
• Commitment to your agenda. Researchers, particularly those attached to universities, may have their own reasons for embarking on an evaluation. It may fit into a doctoral dissertation, a book that a professor is writing, or a piece of long-term research that will eventually be published. Researchers may also have strong prejudices about the kind of research methods they want to use or what they expect to find. You may want to discuss these possibilities up front and specify in your contract that the evaluator will make your program’s needs a priority. Keep in mind, however, that an evaluator with a strong agenda of his or her own may actually prove to be more dedicated to your study and/or work for less money. Just make sure that your agendas, if not the same, are complementary.
  
  
• Data ownership and control. People almost never discuss this issue when they hire an evaluator and often regret the oversight. Insist that the evaluator obtain your prior approval for all public dissemination of results. If you don’t, your evaluator has license to write articles about your program or make conference presentations on the data without your knowledge or approval.
  
  
• Chain of command. It is important to establish that the evaluator will be working for you, not the funder. This means that you will be apprised of all communication that occurs between the evaluator and funder, have the opportunity to review, in advance, all evaluation reports and presentations, and be involved in communicating the findings. Just imagine if your evaluation results turn out to be negative: Wouldn’t you want to have the opportunity to explain why this might have happened or how you intend to improve your program?
  
  
• References. Make sure to ask for references and check them. People almost always do this when they hire staff, but sometimes forget when looking to hire a potential evaluator.

Managing the Logistics
Productive interviews require planning. Small details, like forgetting to reserve an interview room or confirm interview dates with hiring committee members, can quickly turn what might have been a great interview into a fiasco. More importantly, individuals involved in conducting the interviews should be prepared: they should know who they are interviewing, what they hope to find out, and what their roles in the interview process will be. The more prepared the interviewers, the more likely that the interview will be comfortable and informative, and that candidates will feel respected and excited by the prospect of evaluating your program.

There are a number of things you can do before, during, and after the interview to ensure that the process runs smoothly and produces the information you need to make an informed hiring decision.

Finally, keep in mind that an important part of an evaluator’s job is to assist in building the skills, knowledge, and abilities of other staff and stakeholders. It is therefore critical that all invested parties will be able to work well together. Make sure to invite finalists to meet project staff and others with whom they will be working closely to see who best fits with individual styles and your organizational culture. If the fit is good, your evaluation is off to a great start!

Please proceed to Activity 3: Selecting an Evaluator.

References

Center for Substance Abuse Prevention (n.d.). How to find and work with an evaluator. Evaluation Basics PreventionDSS 3.0. Rockville, MD: U.S. SAMHSA/CSAP'’s Center for Substance Abuse Prevention.

Center for Substance Abuse Prevention (2003). Achieving outcomes: A practitioner’s guide to effective prevention (conference edition). Rockville, MD: U.S.

Harding, W. (2000). Locating, hiring, and managing an evaluator. Newton, MA: CSAP's Northeast CAPT.

Juvenile Justice Evaluation Center. (n.d.) Hiring and working with an evaluator. Washington, DC: Author. Available online at
http://www.jrsa.org/pubs/juv-justice/evaluator.pdf.

W. K. Kellogg Foundation (1998). Evaluation handbook. Battle Creek, MI: Author. Available online at http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf.

Rabinowitz, P. (n.d.). Choosing evaluators. Community Tool Box. Available online at http://ctb.ku.edu/tools/en/section_1351.htm.