This guide is for new and potential Service to Science participants, Single State Agency (SSA) and National Prevention Network (NPN) representatives, and other stakeholders interested in learning more about the Substance Abuse and Mental Health Services Administration's (SAMHSA's) Service to Science Initiative. The guide is organized around a list of questions and answers frequently asked about this initiative. If you have any further questions, please do not hesitate to ask! Contact information is provided at the end of this document.
What is SAMHSA's Service to Science Initiative?
Service to Science is a national initiative for locally-developed and innovative prevention programs interested in demonstrating more credible evidence of effectiveness. Established in 2005 by SAMHSA, the initiative's long-range purpose is two-fold:
- To increase the number and array of effective prevention interventions from which states and communities can select to address substance abuse and related mental and behavioral health problems or the underlying factors associated with increased risk
- To support innovative local interventions seeking to strengthen their capacity to demonstrate and document evidence of effectiveness
Service to Science consists of a combination of training events and customized technical assistance (TA). Each year, SSA and NPN representatives nominate a new cohort of participating programs. Pending availability of FY2014 funds, selected programs are invited to attend a three-part Evaluation Primer Webinar Series and participate in an on-site consultation (see below). Programs may then request up to 30 hours of TA from our expert evaluation TA providers between December 2013 and July of 2014.
Who implements the Service to Science Initiative?
Service to Science is implemented by SAMHSA's Center for the Application of Prevention Technologies (CAPT). SAMHSA's CAPT provides training and TA to states, tribes, jurisdictions, and communities across the United States to prevent and reduce substance abuse and related mental, emotional, and behavioral health problems across the lifespan. Service to Science training and TA events are offered by the CAPT's five Resource Teams, located in each of the CAPT's five Service Areas (see the map at the end of this document: "SAMHSA's Service to Science Initiative Contact Information").
What does the Service to Science year look like?
SSA and NPN representatives nominate up to five programs in each state for participation in Service to Science. Programs interested in applying should contact their state's SSA or NPN and ask to be considered for nomination. Programs may also contact the appropriate CAPT Service to Science lead in their service area, whose contact information is provided at the end of this guide. Nominated programs are then invited to submit applications, which are scored according to a set of criteria and priorities that help to determine which programs are most appropriate and best fit for the initiative. Programs selected to participate in the Service to Science initiative are matched with expert evaluation TA providers who work with program representatives throughout the year.
The Service to Science TA phase begins with participation in a three-part Evaluation Primer Webinar series, which outlines some of the pathways to demonstrating evidence of effectiveness and helps program representatives to begin planning for their evaluation. Selected programs are then assigned an evaluation TA provider. This evaluation expert works with program representatives throughout the Service to Science process, starting with the pre-consultation assessment and through to the end of the follow-up TA period. Evaluation TA providers are matched to programs based on program needs, TA providers' areas of expertise, geographic considerations, and availability.
Nomination and Application Process
Who nominates programs for participation in Service to Science?
SSA and NPN representatives nominate programs for participation in Service to Science. Each year, SAMHSA asks each state to nominate up to five programs to participate in the initiative. Programs interested in applying should contact their state's SSA or NPN and ask to be considered for nomination. Programs may also contact the appropriate CAPT Service to Science Lead in their service area, whose contact information is provided at the end of this guide.
What kinds of programs are eligible for participation?
Programs eligible for Service to Science are locally developed, innovative, responsive, practice-based, mindful, and committed. Programs should meet all of the following criteria:
- Innovative in design or focus: These include new services, programs, practices, or policies that are distinguished by their creativity, originality, and utility. Innovative programs do not include existing evidence-based programs or combinations of existing evidence-based programs. However, existing evidence-based programs significantly adapted for specific population groups or substance abuse problems with which they were not originally tested also qualify as innovative. Programs must also be first-time recipients of Service to Science services.
- Responsive to local needs: These are programs that address local substance abuse prevention and related mental health needs (and gaps in service to address those needs), as identified through the application of SAMHSA's Strategic Prevention Framework.
- Fills gaps in the prevention evidence base: These are programs that address substance abuse risk and protective factors and/or problems for which few or no evidence-based prevention interventions have been developed. Examples include, but are not limited to:
- Preventing or reducing adverse childhood experiences and related consequences
- Addressing parental rejection and homelessness among lesbian, gay, bisexual, transgender, and questioning youth
- Incorporating cultural practices and traditions into prevention programming
- Integrating mental health promotion with substance abuse prevention for high-risk populations (e.g., those who have experienced trauma)
- Preventing prescription drug abuse and misuse
- Preventing or reducing high-risk or problem drinking among adults (especially those not attending college)
- Preventing or reducing marijuana misuse and abuse (in light of recent state laws decriminalizing marijuana use in general and for medical purposes)
- Informed by practical experience: These include programs developed with or informed by input from the target population and/or based on practical experience working with the target population in the setting in which the program is delivered. When organizations apply to Service to Science, they must demonstrate that they are rooted in the community they serve.
- Focused on alleviating behavioral health disparities: These are programs that address, via culturally focused or other responsive strategies, disparities in access, quality, and outcomes of prevention programming for vulnerable populations that historically have been underserved or inappropriately served by the behavioral health system. Examples may include:
- Specific racial and ethnic groups (i.e., African Americans, American Indians and Alaska Natives, Native Hawaiian and Pacific Islander youth, and Latina youth)
- Lesbian, gay, bisexual, transgender, and questioning individuals
- Individuals with disabilities
- Women and girls
- Transition-age youth
- Military service members and their families
- Individuals who face economic hardship or live in health care workforce shortage areas
- Committed to evaluation: Programs must be willing to dedicate the effort and time required to enhancing the rigor of program evaluation. Programs must have a funding level sufficient to operate for at least one year and be ready and eager to build evaluation capacity. Based on our experience, programs that commit time to the Service to Science process are more likely to achieve progress in building their evaluation capacity.
How do SAMHSA priorities factor into program selection?
In addition to the above criteria, SAMHSA is interested in programs that address Goals 1 through 4 of SAMHSA's Strategic Initiative #1: Prevention of Substance Abuse and Mental Illness:
- With primary prevention as the focus, build emotional health, prevent or delay onset of, and mitigate symptoms and complications from substance abuse and mental illness.
- Prevent or reduce consequences of underage drinking and adult problem drinking.
- Prevent suicides and attempted suicides among populations at high risk, especially military families; lesbian, gay, bisexual, transgender, and questioning youth; and American Indians and Alaska Natives.
- Reduce prescription drug misuse and abuse.
For more information on the SAMHSA Strategic Initiatives, see: http://store.samhsa.gov/shin/content/SMA11-4629/01-FullDocument.pdf
How can programs apply to participate in Service to Science?
Programs nominated for participation will be asked to complete a Service to Science application. The application includes two parts: (1) a short form to collect contact and program information, and (2) a narrative section that asks applicants to provide the following information on which they will be scored (with the exception of their program evaluation):
- Program rationale: A description of the general problem(s) the program addresses; factors that contribute to the problem(s) as well as its consequences; how the program addresses contributing factors or consequences identified; gaps in services and programming for the identified problem(s); and the program's anticipated substance abuse, and other relevant, outcomes.
- Core program elements: The program's essential approach, strategies, methods, products or practices delivered, as well as when, where, why, and to whom these elements are delivered.
- Program innovativeness: A description of how the program represents a new service, approach, practice, or policy that has not been implemented by others in the field; how it was developed based on work or experience in the field; how it is distinguished from others by its creativity; how it is distinguished from others in terms of its originality; and how it is distinguished from others in terms of its utility or feasibility of implementation. Alternatively, if the program is an adaptation of an existing evidence-based program for a specific population group or substance abuse or related behavioral or mental health problem, how it is new or has not been tried by others in the field; how it was developed based on work or experience in the field; how it is suitable to the target population; and the degree to which it is feasible to implement.
- Program management: A description of how the program is implemented (i.e., organization capacity), who is responsible for implementation, the organizational structures and mechanisms that support implementation, standards for ensuring compliance, communication processes, and systems for overseeing program activities.
- Program evaluation efforts: Information about any current evaluation efforts (e.g., evaluation designs, frequency of evaluation activities, personnel responsibilities, challenges and successes) and organizational support for evaluative learning. Note that this section is for information purposes only and will not be scored.
- Service to Science TA plan: A description of how the program plans to use the evaluation TA available through Service to Science to strengthen its evaluation capacity.
Programs should submit their completed application to SAMHSA's CAPT (guidance on submission is provided in the Service to Science application, which is provided to all nominated programs). The CAPT will review all the applications it receives and select participants based on how well their application scores. Approximately one month after submitting their applications, programs will receive notification of their status—accepted or not accepted. Accepted applicants will also receive contact information for their assigned evaluation TA provider(s).
Programs considered ineligible are those that have already demonstrated evidence of effectiveness in preventing substance abuse outcomes; participated in previous Service to Science training and TA events; or lack funding levels sufficient to operate for at least one year.
The Service to Science Evaluation Primer Webinar Series
What is the purpose of the webinar series?
The Evaluation Primer Webinar Series is the first step in the Service to Science journey. The webinar series is designed to help program representatives begin the process of evaluating their prevention activities and learn more about the Service to Science initiative. It will encourage them to think through ways to make evaluation more feasible and useful, consider evaluation issues or gaps worth addressing, and clarify the questions that they may want to explore with the evaluation TA provider who has been assigned to work with them. The webinars cover the following topics:
- Understanding pathways to becoming an evidence-based program
- Using a five-step framework to guide evaluation planning
- Crafting logic models to explain program and evaluation efforts
When does the Evaluation Primer Webinar Series take place?
The three-part Evaluation Primer Webinar Series (1.5-hour webinars over the course of three days) will be repeated three times between November 2013 and January 2014, pending availability of FY2014 funds.
Who should participate in the webinar series?
Programs should select representatives to participate in the Evaluation Primer Webinar Series who are responsible for developing the program, implementing the program, and evaluating the program. SSA and NPN representatives who nominated programs for participation in Service to Science are not required to participate in the webinar series—however, we encourage them to do so. In some cases, programs may have already identified, or are currently working with, an evaluator. Note that the material covered may be too basic for such evaluators, but they may want to review the content separately.
What happens at the on-site consultation?
Following participation in the Evaluation Primer Webinar Series, programs participate in an on-site consultation with their assigned evaluation TA provider. During the on-site consultation, the program's evaluation TA provider visits the program's home organization and meets with program representatives and stakeholders. The purpose of the consultation is to identify key areas of program need, answer questions about the current status of program evaluation plans, and provide targeted TA to address specific evaluation concerns or issues. Three overarching questions typically guide the discussion:
- Is the need for the program apparent and clearly stated? Do program representatives make a case that the program is needed based on research evidence or existing explanatory theories? Does the program fill a gap in evidence-based prevention programming?
- Are program components and mechanisms clearly explained? Can program representatives explain why the various components of the program were developed and implemented, and how these components are likely to abate the problem(s) or bolster the asset(s) the program is meant to address? Is the program described in such a way that someone else could obtain the same results?
- How do we know the program works? Can program representatives explain what would have happened to subjects in the absence of the program? Does the program account for the anticipated change in outcomes of interest or is it possible that some other factors might account for the change? Consider, for example, the reliability and validity of measures, bias introduced through attrition or selection into the program or comparison condition, appropriateness of analytical strategies, and the ability to account for potential confounders.
At the end of the on-site consultation, the evaluation TA provider will prepare an action plan that summarizes key highlights from the meeting and identifies follow-up action steps agreed to by all attendees.
When and where do on-site consultations take place?
Evaluation TA providers and program representatives collaborate to determine the date and location of the on-site consultation. Soon after notification of acceptance to Service to Science, the program's assigned primary evaluation TA provider will contact the program to schedule the consultation. Pending availability of FY2014 funds, on-site consultations should take place between December 2013 and March 2014. Consultations typically last between 8 and 16 hours (up to two days), depending on need and readiness to address evaluation issues.
Who should attend the on-site consultations?
Programs are responsible for identifying approximately five representatives to participate in the on-site consultations—any more than that makes evaluation planning difficult. We recommend that they include those who developed the program, who are responsible for implementing the program, and who are responsible for evaluating the program. Other stakeholders may also be invited. Additionally, other evaluation TA providers or CAPT staff may participate in the on-site consultation by phone. SSA and NPN representatives who nominated programs for participation in Service to Science are not required to participate in the on-site consultations—however, we encourage them to do so.
How can programs prepare for the on-site consultation?
In addition to the materials submitted as part of their applications (which the evaluation TA providers will have in advance of the on-site consultation), program representatives should feel free to bring any other materials that they think would be helpful, such as a logic model, a description of the program's evaluation design, instruments (e.g., surveys, interview questions, focus group protocols), evaluation results, or evaluation reports. These materials can also be sent to the evaluation TA provider prior to the meeting.
Will the consultation follow a set agenda?
Evaluation TA providers will work with program representatives before the on-site consultation to develop an agenda that will guide the meeting discussion. These agendas typically include opportunities to discuss:
- The purpose of the Service to Science Initiative and the on-site consultation
- Different paths for obtaining evidence-based recognition
- Program rationale
- Program implementation
- Program's evaluation strengths and gaps
- Strategies for meeting those gaps
- Action plan for implementing evaluation strategies
After the On-site Consultations
Will there be any TA available after the on-site consultation?
After the on-site consultations, programs are entitled to up to 30 hours of follow-up TA. During the on-site consultations, programs will work with their evaluation TA providers to develop an action plan that will guide the follow-up TA they will receive. Follow-up TA is a key element of the Service to Science process and is meant to continue the learning community experience after the on-site consultations. Therefore, programs are encouraged to take advantage of the follow-up TA offered to them. Pending availability of FY2014 funds, programs can access this TA following the on-site consultation and until July 31, 2014.
What do programs need to do to receive follow-up TA?
To receive follow-up TA, programs can contact their assigned TA Providers directly. If program representatives have any questions or concerns about follow-up TA, they also should feel free to contact their Service to Science Lead (see contact information below). In addition, TA Providers will contact their assigned programs at fairly regular intervals to assess program progress on or desire to address items in their action plans.
What type of help does Service to Science provide?
Through follow-up TA, programs receive various types of assistance from their assigned evaluation TA providers to meet each program's specific needs. Examples of topic areas for TA include:
- Articulating the rationale or theory of change of the program
- Designing a program evaluation plan
- Developing or identifying existing measures of outcomes of interest
- Analyzing data collected for evaluation purposes
What type of help does Service to Science not provide?
Although evaluation TA providers are prepared to help with program evaluation, they cannot actually conduct program evaluations or prepare materials as part of the evaluation process. For example, they do not collect program data, summarize evaluation results, or prepare evaluation reports. They are able to review and comment on materials, instruments, or reports.
Will the results of STS participants' TA be reported to anyone?
Evaluation TA providers report to the CAPT Service to Science leads on the types of follow-up TA they provide to their assigned programs. This information is also shared with the nominating entity (SSA or NPN) to provide them with information about program progress. In addition, SAMHSA's CAPT evaluators will contact program participants approximately 15 months after selection into Service to Science requesting information about the progress of their evaluation capacity-building efforts. However, SAMHSA will not ask for participant data collected as part of evaluation efforts.
For information about Service to Science, please contact the Service to Science Lead in your CAPT service area:
CAPT Chief, SAMHSA's Service to Science Initiative
Service to Science Lead
SAMHSA's CAPT Southeast Resource Team
Service to Science Lead
SAMHSA's CAPT Central Resource Team
Service to Science Lead
SAMHSA's CAPT Southwest Resource Team
Service to Science Lead
SAMHSA's CAPT Northeast Resource Team
Service to Science Lead
SAMHSA's CAPT West Resource Team