This fact sheet includes common challenges and potential solutions when evaluating environmental change strategies.
Individual change strategies focus on helping people develop the knowledge, attitudes, and skills they need to change their behavior. Environmental change strategies focus on creating an environment that makes it easier for people to act in healthy ways. They do this by changing the culture and contexts within which decisions are made; influencing the community standards, institutions, or structures that shape individuals’ behaviors.
Environmental prevention strategies focus on creating system-level change. They emphasize a broad approach to prevention, associating substance use behavior with not only personal characteristics, but also with environmental influences such as the rules and regulations of social institutions; media messages; and accessibility of alcohol, tobacco, and illicit drugs. Specific examples of environmental change strategies that target substance use include:
Environmental change strategies have specific advantages over strategies that focus exclusively on the individual. Because they target a much broader audience, they have the potential to produce widespread changes in behavior at the population-level. When implemented effectively, they can create shifts in both individual attitudes and community norms that can have long-term, substantial effects: a more cost-effective solution. Ideally, the most effective prevention approaches are those that include a blend of environmental strategies that align with and reinforce prevention strategies directed at individuals.
But evaluating the success of environmental change strategies can be challenging; it requires a shift in thinking–from the individual as the unit of analysis to the population. So, for example, when evaluating the effects of sobriety checkpoints on rates of drunk driving, the unit of analysis would be the population of drivers passing the checkpoints, not the individual drivers. But assessing population-level change isn’t always easy. It often necessitates thinking “outside of the box” and exploring new, less traditional ways to monitor and evaluate prevention programs.
Below are some common challenges (with solutions!) to evaluating environmental change strategies. The list includes challenges related to both process and outcome evaluation.
Process evaluation looks at how program activities are delivered. It helps practitioners determine the degree to which an intervention was implemented as planned and the extent to which it reached the targeted participants. Process evaluation can be used to monitor and document program operations, answering questions such as “Who received services?” “What type of services did they receive?” and “How much or how long did they receive these services?”
Challenge: Measuring Participation. When evaluating individual-level program processes, evaluators often take attendance to measure participation and/or estimate participant completion rates. But measuring participation in an environmental strategy is less straightforward. For example, when implementing a new smoking ordinance in a city, how do you determine the number of people impacted by the ordinance? How do you know who to count—everyone who lives in a city? Only people who go to bars and restaurants? Only individuals who smoke? And how long they should be counted—6 months? One year? Five years?
Solution: To address this challenge, it’s helpful to move away from counting participants, as it doesn’t tell the whole story behind the process. Instead, consider documenting processes in ways that more readily describe how the intervention is, or is not, working. Appropriate methods might include interviews, documentaries, PhotoVoice/participatory photography, and digital story telling (i.e., using media such as audio, music, narratives, visual images, and photographs to capture a unique story).
Challenge: Measuring Fidelity. Fidelity refers to the degree to which a program is implemented as its developer intended. The higher the fidelity of implementation, the greater the likelihood that a replicated program or intervention will have an impact similar to that found in the setting(s) where it was first implemented or tested.
Implementing environmental change strategies with fidelity can be challenging for several reasons. First, while programs targeting individuals often come with (at a minimum) a list of key components or core elements, environmental change strategies rarely come with instruction manuals. Thus, it’s not always clear which elements of the strategy must be maintained in order for the strategy to achieve its expected outcomes. Stated even more plainly, it’s hard to know what an environmental strategy implemented with fidelity even looks like. Moreover, since all communities are different, it’s impossible to expect that any environmental change strategy—even when its core elements are clear—could be implemented in the same way across multiple communities.
Solution: Do your research. See who has implemented the strategy you’re interested in and where it’s been implemented most successfully. Then, from among these successful communities, select the one that is most similar to your own and use it as a model. Talk to people. Find out which elements community members thought contributed most to the program’s success. Then replicate, to the best of your ability, what they did. Adopt the language they used. Work with the same group of stakeholders. Granted, you may not be able to implement the strategy in exactly the same way, but better to model your approach on a similar (and successful!) community than on one that is nothing like your own.
Outcome evaluation measures a program’s results, determining whether a program or strategy produced the desired changes it intended to achieve. Since many environmental prevention strategies either rely on policy change or function like policies, a promising approach to evaluating the effectiveness of strategies is through policy analysis, which takes a broad approach to examining the overall impact of a policy on large populations such as communities, states, or the nation.
Challenge: Assessing the Impact of Individual “Program” Components. Unlike individual change strategies, environmental change strategies often comprise multiple interventions (e.g., a social norming campaign and keg registration), each targeting its own set of risk factors or intervening variables. This can make it very difficult to distinguish which intervention(s) are generating observed changes in a population.
Solution: Focus more on the success of the model, as a whole, than on individual variables. Consider the big picture: Is the model producing successful results? While it’s helpful to understand how each component works, sometimes (especially within the confines of small, low-budget grants) it’s acceptable to just determine that the model works—without knowing exactly why.
Challenge: Collecting Data from All Members of the “Population Group.” Collecting data from an entire population can be an overwhelming—and sometimes impossible—task, depending on the size of the population. Also, gathering data on hard-to-reach population groups, such as young adults between the ages of 18-21 who are out of high school and not attending college, is particularly difficult.
Solution: See what’s already out there. Look for existing data sources that describe behaviors of interest for your population group and those that may capture changes in these behaviors. Common sources of existing national-, state-, or community-level data include:
Challenge: Choosing Research Designs. Choosing an evaluation design for environmental change strategies is challenging for several reasons. First, because the unit of analysis is the population, it’s difficult to implement any of the designs typically used for program evaluation, which involve either assessing change among individuals before and after an intervention, or comparing an intervention/treatment group to a comparison group. Moreover, the process of implementing an environmental change strategy is often more fluid and less circumscribed than the process of implementing an individual change strategy. Policies, for example, are typically embedded in national, economic, political, cultural, and social structures. There are often multiple steps involved in approving and implementing them and implementation can sometimes take months or even years. In addition, the effects of the policy may not be immediate, gradually occurring over time. This makes it difficult to even establish when “pre” ends and “post” begins.
Solution: Look at trends over time, for example, by using an interrupted time series design. This quasi-experimental design looks at trends over time, both before and after an intervention is implemented, and determines at what point the trend is interrupted (e.g., where the observed change occurred). You can then determine if the observed change in the trend came after the intervention was implemented. You can also use a time series design with a control or comparison group (e.g., another similar community), which will allow you to can capture any changes, over time, in the community that received the intervention versus the community that did not receive the intervention.
Challenge: Selecting Comparison Groups/Communities. The addition of a comparison group/community helps you determine whether your target group/population would have improved over time even if it had not experienced your intervention. The more similar the two groups are, the more confident you can be that your program contributed to any detected changes. But what does it mean to be similar?
Solution: No two communities are exactly the same. What’s important is that the intervention and comparison communities are similar with respect to variables that may affect program outcomes (e.g., gender, race or ethnicity, socioeconomic status, education). The more similar the two groups are with respect to these variables, the more confident you can be that your intervention contributed to any detected changes.
When selecting comparison groups/communities, it’s helpful to consider the following questions:
Wambeam, R. A. (April 2013). Challenges and Approaches to Evaluating Environmental Change Strategies. SAMHSA’s Center for the Application of Prevention Technologies Service to Science Team presentation.