Process evaluation looks at how program activities are delivered. It helps practitioners determine the degree to which an intervention was implemented as planned and the extent to which it reached the targeted participants. We all know that implementation quality is critical to maximizing the intended benefits and demonstrating strategy effectiveness. Process evaluation provides us with the tools to monitor quality. It also provides the information needed to make adjustments to strategy implementation in order to strengthen effectiveness.

Specifically, process evaluation can be used to:

• Paint a clear and compelling picture of the population targeted with each strategy
• Reach important target audiences of stakeholders
• Provide data for program improvement efforts
• Distribute the information through as many channels as possible to reach target audience

Some central questions posed in process evaluation are: Who delivers the program? How often? To what extent was the program implemented as planned? How is the program received by the target group and program staff? What are barriers to program delivery? Was the data used to make program improvements/refinements? If so, what changes were made?

Why Not Just Look at Results?

Outcome evaluation looks at results. It measures the direct effects of program activities on targeted recipients, such as the degree to which a program increased knowledge about alcohol and other drug use. But results don’t tell the whole story. Evaluation that only focuses on outcomes is sometimes called a “black box” evaluation because it does not take process evaluation into consideration. Disappointing outcome evaluation results can frequently be illuminated by examining how the program was implemented, the number of clients served, dropout rates, and how clients experienced the program. These are process evaluation questions. Positive evaluation results can also be explained: You can’t take credit for results if you can’t show that you did something that caused them to happen. Outcome evaluation only, without a process evaluation component, won’t provide information about why a program did or didn’t work.

Defining Quality

Quality implementation means that the implementers of each strategy have:

• Assured the strategy matches the cultural, developmental, and gender characteristics of the population
• Received training or technical assistance to support appropriate implementation of the intervention
• Worked with the program developer, policy expert, or the evaluator to understand core components—the elements most responsible for demonstrated outcomes
• Assessed the need for any adaptations to the strategy, especially core components, in order to meet the particular needs of the target population
• Sought input from the program developer about planned adaptations to assure they are consistent with the core components
• Planned necessary adaptations to target population, program content or materials, delivery setting or timeframe to assure integrity of implementation
• Sought to deliver program’s core components with fidelity when possible
• Tracked implementation through process evaluation as well as all planned and unanticipated adaptations to inform outcome evaluation findings
• Utilized process evaluation data to inform and strengthen implementation when outcome evaluation did not reveal desired program results

Process evaluation measures should be designed to assess the degree to which implementers adhered to the items listed above.

What to Do with Disappointing Process Data

If process data suggest the program was not effective, practitioners should work with project stakeholders to determine:

• If the program was implemented with quality
• If strategy was appropriate for the population’s needs (review needs assessment data and program theory)
• Review evaluation strategies and measures to ensure they are appropriate and valid.

Developed under the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for the Application of Prevention Technologies contract. Reference #HHSS277200800004C. For training and/or technical assistance purposes only.