The
following list of annotated resources is organized into the following
sections:
There is also a list of Additional
Online Guides (not annotated).
Program Evaluation Guides
Center for Substance Abuse Prevention (June 2003).
Pathways to effective programs and positive outcomes. Rockville,
MD: U.S. SAMHSA/CSAP’s National Center for the
Advancement of Prevention and Centers for the Application of Prevention
Technologies.
Available online at http://captus.samhsa.gov/southwest/resources/documents/pathways_8-03.pdf.
Developed in response to requests from the prevention
field for guidance in selecting and implementing science-based
prevention programs, this guide presents a capacity-building framework
and process for demonstrating and documenting prevention outcomes.
It contains chapters on needs and assets assessment, capacity-building,
program selection, implementation and assessment, and final evaluation.
Its goal is not to turn readers into expert evaluators, but into
educated consumers who can work comfortably and confidently with
content experts.
Andrews, F. M., Lem, L., Davidson, T. N., O'Malley,
P., and Rodgers, W. L. (1978). A guide for selecting statistical
techniques for analyzing social science data. Ann Arbor, MI:
Survey Research Center, Institute for Social Research, University
of Michigan.
This guide uses decision trees to map the choices
involved in selecting an appropriate statistical technique for
a given analysis. More than 100 different statistics or statistical
techniques are included in the guide. Some knowledge of statistics
is assumed.
Bayer, A. H., Brisbane, F. L., and Ramirez, A. (Eds.)
(1996). Advanced methodological issues in culturally competent
evaluation for substance abuse prevention (CSAP Cultural Competence
Series, 6; DHHS Publication No. SMA 96-3110). Rockville, MD: Substance
Abuse and Mental Health Services Administration.
This publication represents the sixth monograph
in a series of cultural competence publications sponsored by the
Center for Substance Abuse Prevention. It addresses and synthesizes
the complex methodological issues involved in evaluation programs
within multicultural contexts. Chapters develop a framework and
provide suggestions for evaluators who wish to use state-of-the-art
methodological techniques to conduct culturally competent program
evaluations. This book consists of eight complementary chapters
addressing critical methodological issues in program evaluation
within culturally diverse settings.
Brito, C. S., Stewart, K., and Reynolds, R. I. (1998).
Prevention statistics made easy: Understanding correlation,
explained variance and causation. Paper delivered at “Prevention
Planning for Youth Substance Abuse Initiatives,” 11th Annual
National Prevention Network Research Conference, San Antonio, Texas.
This brief paper provides an overview of four basic
statistical concepts: correlation, statistical significance, explained
variance, and causation. It is intended to help practitioners
identify which prevention findings are worth further consideration
and which indicate weaker results.
Carmona, M. C., Stewart, K., Gottfredson, D. C., and
Gottfredson, G. D. (1998). A guide for evaluating prevention
effectiveness, CSAP technical report (NCADI Publication No.
98-3237). Rockville, MD: U.S. Department of Health and Human Services,
Substance Abuse and Mental Health Services Administration, Center
for Substance Abuse Prevention.
This guide provides practitioners with basic evaluation
concepts and tools. It describes commonly used research designs
and their strengths and weaknesses. It also describes qualitative
and quantitative data collection methods used in process and outcome
evaluation and discusses basic concepts in data analysis.
Fetterman, D. F., Kaftarian, S. J., and Wandersman,
A. (Eds.) (1996). Empowerment evaluation: Knowledge and tools
for self-assessment and accountability. Thousand Oaks, CA:
Sage.
This publication explores empowerment evaluation,
a method for using evaluation concepts, techniques, and findings
to foster improvement and self-determination. Program participants—including
clients—conduct their own evaluations, with an outside evaluator
often serving as a coach or additional facilitator, depending
on internal program capabilities. The text focuses on the various
contexts in which empowerment evaluation is conducted, focusing
particularly on the role empowerment evaluation has played in
the U.S. Department of Health and Human Services’ substance
abuse prevention programs. The contributors provide tools and
technical assistance for implementing this approach, as well as
ideas for strengthening the links between empowerment evaluation
and community capacity building.
French, J. F. and Kaufman, N. J. (Eds.) (1981). Handbook
for prevention evaluation: Prevention evaluation guidelines
(Publication No. ADM81-1145). Washington, DC: National Institute
on Drug Abuse.
This handbook was written for evaluator-practitioner
teams working to apply their skills in the assessment and improvement
of prevention programs. Topics discussed include models of prevention,
evaluation design, indicators and measures for process and outcome
evaluation, and reporting evaluation results. An extensive appendix
on instruments and data sources is included.
Hawkins, J. D. and Nederhood, B. (1987). Handbook
for evaluating drug and alcohol prevention programs: Staff/team
evaluation of prevention programs (Publication No. (ADM) 87-1512).
Washington, DC: U.S. Department of Health and Human Services.
This handbook provides program managers with a comprehensive
tool to guide their evaluation efforts. It discusses instruments
and activities for determining program effectiveness (outcome
evaluation), and for documenting and monitoring the delivery of
services (process evaluation). Major topics discussed include
evaluation design, measuring outcomes, measuring implementation,
data collection, data analysis, and reporting study findings.
Worksheets, sample instruments, and a bibliography are included.
Health Canada (August, 1996). Guide to project
evaluation: A Participatory Approach. Ottawa, Ontario: Author.
Available online at
http://www.phac-aspc.gc.ca/ph-sp/resources-ressources/guide/index-eng.php
This evaluation guide provides a comprehensive,
easy-to-use, framework for project evaluation. It is premised
on the belief that evaluation can be a useful and positive experience
that promotes learning and action, and that what is learned from
project evaluation is as important as what the project produces
or creates.
Isaac, S. and Michael, W. B. (1983). Handbook
in research and evaluation: A collection of principles, methods,
and strategies useful in planning, design, and evaluation of studies
in education and the behavioral sciences (2nd ed.). San Diego,
CA: EdLTS.
Intended to help practitioners choose the best technique
for a particular study, this book summarizes basic information
on research and evaluation methods. Topics discussed include planning
evaluation and research studies, research design and methods,
instrumentation and measurement, data analysis, and reporting
a research study. It includes many tables and worksheets.
W. K. Kellogg Foundation. (1998). W. K. Kellogg
Foundation evaluation handbook. Battle Creek, MI: Collateral
Management Company. Available online at http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf
This handbook offers a framework for thinking about
evaluation as a program tool. Written for project directors with
direct responsibility for the evaluation of Kellogg Foundation–funded
projects, it discusses how to prepare for an evaluation (e.g.,
develop evaluation questions, budget for evaluation, select an
evaluator), design and conduct an evaluation, and report findings.
The handbook contains worksheets, charts, and a bibliography on
evaluation.
Kozel N. J. and Sloboda Z. (1998). Assessing
drug abuse within and across communities: Community epidemiology
surveillance networks on drug abuse (NIH Publication No. 98-3614).
Rockville, MD: National Institute on Drug Abuse. Available online
at http://www.drugabuse.gov/Pubs/Assessing/.
This guidebook is meant to help practitioners at
the local, regional, and state level assess local drug abuse patterns
and trends using indicator data. Data sources discussed include
treatment data, medical examiner/coroner data, the Drug Abuse
Warning Network (DAWN) data, law enforcement data, national surveys,
HIV/AIDS data, census data, and telephone hotline data. The guidebook
includes references, a glossary, and appendices that identify
or discuss data sources.
Larson, M. J., Buckley, J. and Gabriel, R. M. (1997).
A community substance abuse indicator's handbook: How do we
know we are making a difference? Boston, MA: Join Together.
This guide identifies key indicators (i.e., information
typically collected by agencies or organizations) that community
coalitions and other groups can use to describe the nature and
scope of local substance abuse problems. It also provides contact
information on state agencies and organizations that collect and/or
report indicator data. This document can be ordered online at
www.indicatorshandbook.org.
A summary of this document can be found in “Beyond Anecdote:
Using Local Indicators to Guide Your Community Strategy to Reduce
Substance Abuse” (1999 Monthly Action Kit, Special 1999
Issue, Boston, MA: Join Together, 1999).
Madison, A. (Ed.) (1992). Minority issues in program
eEvaluation: New directions for program evaluation Series 53.
San Francisco, CA: Jossey-Bass.
This publication focuses on the importance of minority
participation in program evaluation. According to the author,
the text “constitutes a new direction in program evaluation,
in that it links methodological, moral, and ethical evaluation
issues to the minorities who have the highest stake in the attainment
of social policy and program goals.” It contains six articles:
Two address insufficiency and/or risks related to equity issues
and quantitative evaluation, three discuss inclusion and participation
of minority and/or multicultural voices in program evaluation
development, and one focuses on assessing the research on multicultural
education.
Miller, D. C. (1991). Handbook of research design
and social measurement (5th ed.). Newbury, Park, CA: Sage.
This handbook provides procedures and guidance for
three major types of research: basic, applied, and evaluation.
It addresses topics such as research design, data collection,
statistical analysis, and scales and indexes, and includes a guide
to federal and private funding and to the publication of research
reports. Extensive bibliographies accompany each major section
of the handbook.
Moberg, D. P. (1984). Evaluation of prevention
programs: A basic guide for practitioners. Madison, WI: Board
of Regents of the University of Wisconsin System for the Wisconsin
Clearinghouse.
This guide, which provides detailed steps for planning
and implementing a program evaluation, is intended for practitioners
involved in planning and delivering local prevention services.
Muraskin, L. D. (1993). Understanding evaluation:
The way to better prevention programs (Publication #ED/OESE92-41).
Washington, DC: U.S. Department of Education.
This handbook was written for school and community
agency staff to carry out required evaluations under the Drug-Free
Schools and Communities Act. The premise of this book is that
many evaluations that use simple designs can be conducted without
formal training in program evaluation (though the author does
identify checkpoints in the evaluation process where practitioners
may want to consult with evaluation specialists). Topics discussed
include evaluation design, data collection methods and instruments,
and interpreting and reporting findings. This publication is available
at: http://www.ed.gov/PDFDocs/handbook.pdf.
Thompson, N. J. and McClintock, H. O. (1998). Demonstrating
your program's worth: A primer on evaluation for programs to prevent
unintentional injury. Atlanta, GA: National Center for Injury
Prevention and Control, Centers for Disease Control and Prevention.
Designed for program managers, this guide describes
the process involved in conducting a simple evaluation (formative,
process, impact, and outcome), how to hire an evaluator, and how
to incorporate evaluation activities into a prevention program.
Appendices include information on sample questionnaire/interview
items, events or activities to observe, and types of records to
maintain, as well as a glossary and bibliography on evaluation.
Information on ordering this publication can be found online at
http:/www.cdc.gov/ncipc/pub-res/demonstr.htm.
U.S. Department of Transportation (1999, May). The
art of appropriate evaluation: A guide for highway safety program
managers (DOT HS 808 894). Washington, DC: U.S. Department
of Transportation, National Highway Traffic Safety Administration.
This guide demystifies what is meant by evaluation,
describes common stumbling blocks, and discusses how to choose
an appropriate evaluation method to promote the hard work that
goes into prevention programs. Though intended for state or local
highway safety project directors, it is a useful manual for all
involved in evaluation planning and implementation. A background
in experimental design or statistics is not required—the
guide tells where you can find experts to help with that part.
Resources Related
to Cultural Competence
Nelson, D. E., Brownson, R. C., Remington, P. L.,
and Parvanta. C. (2002). Communicating Public Health Information
Effectively: A Guide for Practitioners. Washington, DC: American
Public Health Association.
This book is intended to help health practitioners
in both the public and private sector improve their ability to
communicate with different audiences. It provides practical recommendations
and examples on how to communicate public health information to
nonscientific audiences more effectively. It is intended to be
used primarily as a primer to assist those who need background
information or guidance when faced with specific communication
situations.
Sayre, K. (2003). Guidelines and best practices
for culturally competent evaluations. Colorado Trust. Available
online at http://www.coloradotrust.org/attachments/0000/3116/GuidelinesBestPracticesCulturally04.pdf
This report contains many tools and guidelines for
developing and implementing a culturally competent evaluation.
Though intended for health and social science researchers engaged
in evaluations of programs serving ethnically diverse populations,
people who use evaluation results to make programmatic decisions
will also find this information useful.
REFT Institute, Inc. (2003). Keys to cultural
competency: A literature review for evaluators of recent immigrant
and refugee service programs in Colorado. Colorado Trust. Available
online athttp://www.coloradotrust.org/attachments/0000/3182/KeystoCulturalCompetency04.pdf.
This publication details unique characteristics
of nine separate cultures (including people from Bosnia-Herzegovina,
Kurdistan, the former Soviet Union, Central America, Mexico, Laos,
Vietnam, Somalia and Sudan) and related implications for conducting
research and evaluation within these cultures.
Resources Related
to Working with an Evaluator
Budget and Evaluation Division, Financial Management
Board Secretariat, Government of the Northwest Territories, Canada
(1999). Working well with evaluation consultants: A guide.
Available online at http://www.fin.gov.nt.ca/documents/forms-documents/consultantguide.pdf
.
This guide, designed to help people and organizations
who are planning to work with a consultant to conduct an evaluation,
outlines steps to take to ensure an effective partnership and
a useful and credible product.
Juvenile Justice Evaluation Center (n.d.). Hiring
and working with an evaluator. Washington, DC: Author. Available
online at http://www.jrsa.org/pubs/juv-justice/evaluator.pdf.
This short and straightforward guide is one of a
series of briefings designed for program managers.
Rabinowitz, P. (2003). Choosing evaluators.
University of Kansas Work Group on Health Promotion and Community
Development and AHEC/Community Partners. Available online at http://ctb.ku.edu/tools/en/section_1351.htm.
The Community Tool Box Web site provides numerous
how-to tools designed to help practitioners with the different
tasks necessary for community health and development. The section
on choosing an evaluator is part of a larger chapter on program
evaluation.
Online Courses
Evaluation for the Unevaluated: Program Evaluation
101
Center for Substance Abuse Prevention
http://pathwayscourses.samhsa.gov/eval101/eval101_intro_pg1.htm
This is the first in a series of three self-paced
courses developed by CSAP focusing on evaluation. It introduces
key terms and concepts, explains different forms of evaluation,
and provides suggestions on choosing an evaluator, organizing
an evaluation team, and planning an evaluation.
Evaluation for the Unevaluated: Evaluation 102
Center for Substance Abuse Prevention
http://pathwayscourses.samhsa.gov/eval102/eval102_intro_pg1.htm
Picking up where Program Evaluation 101 ended, this
course reviews the planning process and how evaluation fits in,
data analysis and uses, and challenges to program evaluation.
Wading Through the Swamp: Evaluation 201
Center for Substance Abuse Prevention
http://pathwayscourses.samhsa.gov/eval201/eval201_intro_pg1.htm
Building on the two previous courses, Wading
Through the Swamp uses a case study approach to provide prevention
practitioners with a practical understanding of data analysis
and interpretation.
Additional Online Guides
Basic Guide to Program Evaluation
Carter McNamara, M.B.A, Ph.D., Authenticity Consulting, LLC
http://www.managementhelp.org/evaluatn/fnl_eval.htm
Evaluation Cookbook
Learning Technology Dissemination Initiative
http://www.icbl.hw.ac.uk/ltdi/cookbook/contents.html#endhead
Getting to Outcomes: Methods and Tools for Planning,
Self-Evaluation, and
Accountability
Abraham Wandersman, Ph.D. and Pamela Imm, Ph.D., University of South
Carolina; Matthew Chinman, Ph.D., The Consultation Center, Yale
University School of Medicine, and Shakeh Kaftarian, Center for
Substance Abuse Prevention
http://www.rand.org/pubs/technical_reports/TR101/index.html
A Guide to Family Intervention and Prevention
Program Evaluation
Children, Youth, and Families Education and Research Network
http://www.fourh.umn.edu/evaluation/family/default.html
Measuring Program Outcomes: A Practical Approach
United Way of America
http://national.unitedway.org/outcomes/resources/mpo/
Planning a Program Evaluation
University of Wisconsin Cooperative Extension
http://learningstore.uwex.edu/pdf/G3658-1W.PDF
Taking Stock: A Practical Guide to Evaluating
Your Own Programs
Horizon Research, Inc.
http://www.horizon-research.com/reports/1997/taking_stock.php
User-Friendly Handbook for Mixed-Methods Evaluation
National Science Foundation
http://www.nsf.gov/pubs/1997/nsf97153/start.htm
User-Friendly Handbook for Program Evaluation
National Science Foundation
http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_6.pdf
|
