Overview of Evaluation

Evaluation is the systematic collection and analysis of information about interventions to improve effectiveness and make decisions. It is an intrinsic part of all the steps of the SPF.

Evaluation is useful for some of the following reasons:

• Helps to assess the progress of an intervention.
• Identifies what does and does not work in a particular setting.
• Builds community capacity.
• Strengthens accountability.
• Supports sustainability.
• Influences decision-makers.

An evaluation includes both process and outcome evaluation data. Process evaluation occurs during the implementation of an intervention; outcome evaluation occurs after the intervention has been implemented. Collecting this evaluation data will help you do the following:

• Monitor implementation.
• Improve implementation and performance.
• Make future decisions using the data to determine what interventions and outcomes should be sustained.

Process evaluation documents all aspects of the implementation of an intervention (see “Documenting the Process” on Information Sheet 1: Phases of Implementation). For example, was the same material presented in the same number of sessions over the same timeframe using the same methods? Process evaluation answers the question: “Did we do what we said we would do?” This type of evaluation data will help determine the following:

• Were interventions implemented as planned?
• What adaptations were made?
• Were the resources sufficient?
• What obstacles were encountered?
• Who participated and for how long?

Outcome evaluation documents whether the intervention made a difference, and if so, what changed. It documents effects achieved after the intervention is implemented, such as changes in a population group’s knowledge, attitudes, skills, or behavior that the intervention is expected to produce in both the short term and the long term. Outcome evaluation answers the question: “Did our intervention make a difference—did it impact the risk factors/problem we wanted to address?” This type of evaluation data will help to determine the following:

• What changes actually occurred.
• How these changes compare to what the intervention is expected to achieve.
• How these changes compare with those not exposed to the intervention.

Outcomes
A community will need to identify the short-term and long-term outcomes they hope to achieve with their overall comprehensive plan as well as for each intervention. Remember that if the prevention intervention does not address the underlying risk and protective factors that influence the targeted substance abuse problem, then it is unlikely to produce positive outcomes or changes in that problem.

Short-term Outcomes:

• Short-term outcomes are the immediate effects that the intervention is expected to achieve. The outcomes are expressed as changes in knowledge, attitudes, and skills of the focus population at the end of the intervention.
• Short-term outcomes tend to be connected to changes that occur in the risk or protective factor.
• Be aware that how well the intervention is implemented can have an impact on short-term outcomes.

Long-term Outcomes:

• Long-term outcomes are the ultimate effects of the intervention at some point after the intervention is completed, maybe six months or a year later.
• Long-term outcomes depend on the short-term outcomes because short-term changes in knowledge, attitudes, or skills can lead to long-term behavior change.
• Long-term outcomes tend to be connected to the behaviors and related problems that you are trying to change.

Developed under the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for the Application of Prevention Technologies contract. Reference #HHSS277200800004C. For training and/or technical assistance purposes only.