X. Evaluation Tools & Measures
K. Using Community Archival and Indicator Data
Adapted From the Community Tool Box Chapter 31, Section 9
Community-Level Indicators
Contributed by Aimee Wittman
Edited by Bill Berkowitz and Jerry Schultz
What are community-level indicators?
For community initiatives, community-level indicators can be seen as one ultimate outcome of any initiative. While the monitoring system, constituent surveys, goal attainment reports, and interviews can help pinpoint the success of individual events, community-level indicators try to determine how these individual events affect the community as a whole.
One way to discover the big picture effect is to review proven "indicators" recommended by experts in your group's field of interest. For a group focusing on substance abuse, for just one example, one indicator might be the number of emergency medical transports related to alcohol use. By examining these numbers, your group will have a better sense of the ultimate effect of your group's work on the community at large.
Why should you use community-level indicators?
While the other evaluation tools can offer you valuable insights into the effectiveness of specific actions and objectives of the initiative, community-level indicators provide the major "big picture" perspective. But the benefits don't stop there!
Community-level indicators provide "bottom-line" evidence of the impact of the initiative.
"Bottom-line" evidence describes the ultimate effect of your group's work on the initiative. Community-level indicators give you an objective way to measure and assess the bottom-line evidence. Examples might include single-nighttime vehicle crashes (as a way to track efforts to reduce substance abuse) or the number of hospital admissions for violence-related injuries as a way to account for efforts to reduce youth violence). In addition, community-level indicators help you measure progress towards the goal. Measurements are ideally taken at multiple stops along the road, so that you (and the community) can see how you are doing and adjust your course, if need be.
Community-level indicators help determine the effects of key components of the initiative.
If you decided to implement several bold ideas in your action plan, community-level indicators can help you determine the effectiveness of these ideas, and of specific objectives. That way, you can see which ideas and strategies helped you achieve some of your goals, and which might need some reworking in a future action plan.
Community-level indicators help push issues to the forefront of the public agenda.
If the community-level indicators reveal negative results (i.e. the number of single nighttime-vehicle crashes stays the same even after you have implemented several of your objectives), this can be the perfect time to remind the community that the issue still exists. It still needs to be addressed through continued action. Although negative results may not be the results you want to see, try to envision them as a golden opportunity to push your issues to the forefront of the public agenda.
Community-level indicators that show positive results can help secure more support for the initiative.
But, if your community-level indicators suggest that your work has been resulting in many positive changes for the community, you can use these glowing reports to gain more support for your initiative and the good work that you do. This might include financial support, which can be used to improve and extend the work of the initiative. So get out there and tell it to the world; your group deserves recognition and support for all of its hard work and contributions!
When should we evaluate community-level indicators?
The sooner you start to evaluate your community-level indicators, the sooner you'll get a better understanding of how your community is affected by different variants. This way, you can plan in advance so that the effects are the ones that you desire.
It's advisable to start measuring community-level indicators as soon as you start a new project or event, so that you can change things around as necessary. To get a broader picture is always valuable and the sooner the better.
Of course, after you conclude your event or project, run a detailed community-level indicator survey so that you can measure the results and effects of your activity , and learn more for the future.
How do we find community-level indicators?
It's time to dip your toes in the crystal blue waters of community-level indicators. Soon you'll be swimming with the big fish as you navigate the waves of availability, accuracy, feasibility and sensitivity. Let's dive in!
There are five main steps to evaluating community-level indicators. They involve:
Now, let's get our feet wet!
1. Select indicators that are available, accurate, possible to collect, and sensitive to the initiative.
The key to success here starts with knowing the right questions to ask. First, your group will want to hold several meetings with staff and also with groups of community members to identify which community-level indicators will best serve your needs.
In the Examples section, you will find a variety of recommended community-level indicators for:
Let's say, for example, that your group wants to focus on reducing the rate of teenagers who smoke marijuana in your community. One of the community-level indicators includes the number of juvenile arrests for drug offenses. (See Examples )
2. Contact relevant local and state agencies to find data.
Once you have decided upon the appropriate community-level indicators, members of the staff can pool their knowledge of local, regional, state, and federal sources who might be good resources for information. These sources might include health departments, law enforcement agencies, or transportation offices. Your exact choices will be determined by what you want to study.
Let's continue on with the above example. If you know that the community- level indicator you want to measure is the number of juvenile arrests for drug offenses, you could contact the local police department. Then, gather information about the number of arrests for drug offenses for community members between the ages of 14-20 over the last several years.
3. Compile, summarize, and graph the data.
Members of the evaluation team put together all of the crucial information in a report, and also in graph, chart or table form.
How do we graph the data from the number of drug arrests? One way would be to graph the numbers of arrested people at six-month intervals. Then, you can compare your group's presence in the community with the changes in the numbers. If, for example, you began implementing your action steps one year ago, you would want to compare the levels of drug arrests before your work with the levels during and after your presence.
4. Present the data to community leadership, trustees, and funders.
Data obtained from the community-level indicators can then be used to inform and educate your organization and the community. As a tool to encourage public support, community-level indicators can highlight the positive results of your group. Feel free to spread the word far and wide about the results you have obtained. Post your information everywhere; let the community know what is going on in their world.
5. Use the data to elevate the issue on the public agenda, and to redirect the initiative's efforts.
If the community-level indicators yield negative results, this can provide a chance to push the issue to the top of the public agenda. Additionally, data, regardless of the results, can give the group a chance to redirect its work in a more effective direction.
If, for example, your research shows that levels of drug arrests among juveniles continues to hold steady, this would be an excellent opportunity to remind the community that drug abuse exists and continues to be a significant issue in the community. In this case, you might want to distribute your results to local schools, local community leaders, and other groups or organizations that focus on substance abuse.
Finally, remember that results can be viewed in a variety of ways. Positive results might mean that your group needs to redirect its energy towards other areas if you have achieved your original goals. Negative results should not be viewed as failure. Instead, these results should simply mean that another strategy might work better. Then, rather than continuing to focus your energy in the wrong direction, you can try to find a more effective way to work towards change.
To sum up:
You deserve a round of applause! You have successfully worked your way through all of the evaluation steps! Good work! Keep reading to find some helpful examples of community-level indicators. Then, prepare to enter the wonderful world of Feedback! (Want to know more? Check out Chapter 32!)
We encourage the reproduction of this material but ask that you credit the Community Tool Box.
Examples
All examples are from Fawcett, S.B., Paine-Andrews, A., Francisco, V.T., Schultz, J.A., Richter, K.P., Lewis, R.K., Williams, E.L., Harris, K.J., Berkley, J.Y., Fisher, J.L., & Lopez, C.M. (1994). Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development, Lawrence, KS: Work Group on Health Promotion and Community Development, University of Kansas.
Example #1: Recommended Community-Level Indicators for Substance Abuse Coalitions
Example #2: More Recommended Community-Level Indicators for Substance Abuse Coalitions
Example #4: Recommended Community-Level Indicators for Tobacco Control Initiatives
Note: This represents the most objective data available on population levels of tobacco consumption. Data are available from the state health department ( or department of revenue) based on the excise taxes that are imposed on tobacco products. Excise taxes are collected at the level of tobacco distributors. This information is available on a monthly basis.
Variability from month to month may be an artifact of this measure. It may reflect patterns of stocking at the retail level. When systematic seasonal variations are adjusted for, however, collective sales data provide the best available estimate of total tobacco use.
The main limitation of these data is that they do not provide information on the behavior of consumers. A change in consumption rate is a composite of many individuals' uptake and quitting behavior. For example, a drop in cigarette sales may be the result of the same people smoking fewer cigarettes or fewer people smoking. Nor do consumption data indicate what changes in special populations, such as young women or ethnic minorities, might be reflected in the data. Despite these limitations, tobacco consumption remains the best community-level indicator.
Example #6: Recommended Community-Level Indicators for Violence Prevention Initiatives
|
|
