Service to Science Strengthens Evaluation Capacity of Pacific Programs

2011 STS Academy. L to R: Wil Maui, James Arriola, Marie Auyong, Eric Ohlson, Maybelline Ipil, Natalie Nimmer, Joie Heine, Gaafar Uherbelau, Gretchen Casey, Elizabeth Rechebei, Wayne Harding, Cece Kilma, Eric Albers, Annette David

Across the U.S. Pacific Jurisdictions, local prevention leaders develop innovative and culturally appropriate programs and practices to reduce substance abuse and enhance social and emotional health in their communities. But evaluating the effectiveness of these programs can be challenging. As with other indigenous populations, the types of data and data collection methods these communities value may differ from what funders traditionally require. And the geographic remoteness of the jurisdictions has resulted in limited technology and a minimal evaluation workforce.

To help expand and develop program evaluation skills in such communities, five programs in the Pacific Jurisdictions are accepted each year into Service to Science (STS), a national initiative supported by the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention (CSAP). Implemented by the Collaborative for the Application of Prevention Technologies (CAPT), STS helps innovative, field-grown prevention interventions that address substance abuse demonstrate, improve, and document evidence of their effectiveness. According to CAPT Chief of STS Kim Dash, “ STS is an effective model for the Pacific Jurisdictions because it offers a combination of face-to-face and distance technical assistance (TA), tailored to meet the unique evaluation needs and readiness of participating programs.”

On April 4–5, the CAPT West Resource Team held its annual STS Academy for programs in the Pacific Jurisdictions. Representatives from this year’s five programs traveled to Kolonia, Pohnpei, in the Federated States of Micronesia, to meet with STS staff and TA providers to learn more about the STS process and develop action plans for strengthening their own evaluation abilities. The participating programs ranged from a public communication campaign to heighten awareness of the dangers of drunk driving, to a music program that builds on the culturally significant process of playing the ukulele to strengthen relationships between clients in recovery and their family members, to a faith-based project that combines counseling, recreational activities, and tutoring to reduce substance abuse, school dropout rates, and suicide among school-age youth. The five programs hail from four jurisdictions: Guam, Federated States of Micronesia, Republic of the Marshall Islands, and Republic of Palau.

The group discusses evaluation. L to R: STS TA Provider Eric Albers, CAPT Associate Wil Maui and Statistician James Arriola (CNMI), School Counselor Cece Kilma (RMI), MIEPI Financial Officer Joie Heine and MIEPI Project Manager Maybelline Ipil (RMI)

One of the hallmarks of STS is that the TA providers work one-on-one with participating programs, providing customized support to meet individual program needs and goals. This approach is especially important for indigenous programs because it meets them at their current evaluation readiness and capacity, and moves their skills forward while honoring their culture. During the Academy, the program representatives met for eight hours with their assigned TA providers to develop logic models and evaluation plans moving forward. The Academy also offered numerous opportunities for peer sharing, activities designed to enhance evaluation knowledge, and an assessment of individual program strengths and needs across an evaluation framework. Following the Academy, participating programs can receive up to an additional 40 hours of TA to continue the work.

One participant expressed appreciation for her and her colleagues being able to tell their own story on their own terms. “All of this work is so integral in being able to say for ourselves and tell people what works for us, what tool works for us, what doesn’t work for us, and being able to fight for what works for us,” she said.

STS TA Provider Eric Albers (R) provides evaluation technical assistance to Majoro Cooperative High School Principal Natalie Nimmer and School Counselor Cece Kilma (RMI).

An additional component of this year’s Pacific STS Academy was an apprenticeship program designed to build the evaluation workforce residing in the Pacific Jurisdictions. The CAPT West Resource Team strives to pair its Pacific STS programs with local evaluators during the STS Academy, but finding enough trained evaluators in the Pacific to work with STS is difficult. To address this, the CAPT invited Pacific Islanders with experience in collecting, analyzing, and using data to service as apprentice STS TA providers, shadowing seasoned Pacific STS TA providers during the Academy to learn more about the initiative, evaluation fundamentals, and TA provision. After the Academy concluded, the apprentice TA providers met on April 6 with the CAPT West Resource Team staff and the Pacific STS TA providers to reflect and share their observations of the Academy and develop their own evaluation TA skills.

This year’s five Pacific Jurisdiction programs participating in STS are:

DeWill 2 Live, Republic of Palau
Majuro Cooperative School’s Substance Abuse Prevention Program, Republic of the Marshall Islands
School Caravan Project, Republic of the Marshall Islands
Music Matters in Prevention and Recovery, Guam
Sekere Youth Association, Pohnpei, Federated States of Micronesia

To learn more about the Service to Science Initiative, click here to e-mail Kim Dash [1].