In order to address the level of this risk or protective factor:

(goals)

We will do the following program activities

For these people and for this amount of time

(target population)

We expect that this activity will lead to changes in these factors, ________________ which in turn will lead to our program goal

(theory of change)

We will know these changes have occurred if:

(short term outcomes)

We will know we are reaching our goals if:

(long term impacts)

(1) Community laws and norms;

(2)

Favorable attitudes towards drug use

(1) Develop radio and television advertisements to air during after-school times;

(2) Develop billboards to locate in the community;

(3) Place advertisements in youth-oriented magazines

an unspecified number of children and adolescents who see ads. Estimates:

TV ad run in 3000 local homes daily;

magazine circulates once per month to 500 area youth;

billboards will be seen by up to 1000 children daily;

campaign will run for three months.

The program will lead to changes in perceived norms among children and adolescents regarding tobacco use, such that:

Fewer children and youth will believe that smoking is "cool" and acceptable behavior, which will lead to less smoking behavior.

(1) More children and adolescent report that most of their peers don’t smoke, and that smoking isn’t "cool";

(2) More children and adolescents report that smoking tobacco is not a positive behavior;

(3) Heightened awareness of tobacco industry advertising tactics

(1) Rates of tobacco smoking decline among children and youth

Example Evaluation Questions:

How many radio and TV ads were aired?

How many billboards were posted?

How many advertisements were placed in magazines?

How many youth saw the advertisements and billboards?

How often did they see them?

Which type of advertisement was seen by the most youth?

Did youth understand the content of the ads?

Do youth remember the content of the ads?

Was the correct message conveyed about norms?

Which type of advertisement do youth think is most effective in conveying the message?

Were there things about the ads that youth didn’t like?

Did the advertisements lead to changes in youth’ perceptions of norms?

Do youth who are exposed to the ads see smoking as more dangerous and less "cool"?

Are youth more aware of the advertising tactics of the tobacco industry?

Did the program lead to reductions in smoking among youth?

Possible Measures and Sources of Information:

Number of advertisements & billboards implemented.

Number of youth reporting in a survey that they’ve seen the advertisements, billboards, etc.

Focus group with youth done prior to implementation to ensure content is appropriate. Telephone follow-up with these youth to ensure that they remember the message content.

Self-report survey of youth attitudes and perceptions of tobacco use; knowledge of tobacco industry advertising tactics

Rate of tobacco use self-reported by youth

Example Analysis Strategies:

Frequency count

Frequency count of "yes" responses to questions about awareness of advertisements.

Review of analysis of notes taken during focus group session. Summary of major points and suggestions.

Frequency count of the number of youth who remember the content.

Measures taken before and after advertising campaign.

Pre-Post differences.

Measures taken before and after advertising campaign, pre-post differences.

Archival data: change from previous year’s reported data.

In order to address the level of this risk or protective factor:

(goals)

We will do the following program activities

For these people and for this amount of time

(target population)

We expect that this activity will lead to changes in factors, ________________ which in turn will lead to our program goal

(theory of change)

We will know these changes have occurred if: (short term outcomes)

We will know we are reaching our goals if:

(long term impacts)

(1) Friends involved in problem behavior;

(2) Academic failure;

(3) Persistent anti-social behavior;

(4) School bonding

(1) Personal Growth classes will be held daily taught by school counselors. One course will be offered per grade. Classes will include exercises to enhance self-esteem, teach positive decision making and social skills, and communication skills. We will obtain teaching materials from the program designers.

(2) Also, weekly Friday evening movie hours will be held that include students in Personal Growth classes and for any students interested in drug-free activities. Students will watch & discuss movies showing positive teenaged role models.

(3) Individual counseling will be available to students who request it.

Eligible students in High School A (9-12 grades) include all those who show signs of poor school achievement as evidenced by one or more of the following:

*D grades or lower in 2 or more classes

*Missing more than 7 days of school in a month

*Referral for behavior problems

These students will: attend 1 semester Personal Growth class; attend weekly social events; counseling as needed.

Estimate: 50 youth will be served in each grade.

Classes & counseling will enhance self-esteem, social skills, and communication skills. These skills will lead youth to feel better about themselves and this will lead to more positive attitudes towards school. Students will have more motivation for school work and will be less likely to be involved in problem behavior.

[Note that the pathway to improved school performance and ATOD use is indirect, that is, the program is expected to lead to changes in self-esteem and other skills which in turn may lead to improved school outcomes and reduced substance use. Substance abuse issues are not dealt with directly as part of the curriculum]

Social activities will create opportunities for friendships to develop with students not involved with problem behavior, and to develop positive peer role models. These changes in the peer groups of youth will lead them to be less likely to use drugs and alcohol.

Students show increased self-esteem, communication skills, and social skills.

Students report fewer friends who use ATOD.

Students report having more positive peer role models.

Students feel stronger motivation and commitment to school.

Reductions in ATOD use.

Improved School Performance.

Improved school attendance rates.

Reduced Drop-Out rate.

Example Evaluation Questions:

How many students attended each class?

How many classes, on average, did each student attend?

Did the teachers implement the curriculum as intended?

How many students attended the movie social hours? How many of these were also attending the Personal Growth classes?

How many students requested counseling? How many received counseling?

How many students were referred to the program?

Did the referred students meet the eligibility requirements? If not, how were decisions made to refer them to the program?

How do students react to the curriculum? Do they see the classes as helpful?

Do students feel the Personal Growth curriculum helps them to be more motivated about school?

Do students understand the movies shown during the social hours? Do they understand the messages conveyed by the movies?

Do youth who aren’t part of the Personal Growth classes attend the movies? Do they socialize with the other students?

Do students participating in the classes show increased self-esteem, self-confidence and motivation for school?

Do students in the program make new friends who are non-using?

Are students in the program more likely to report that they have a positive role model?

Does the program lead to reductions in ATOD use?

Does the program lead to improved School Performance?

Does the program lead to improved school attendance rates?

Does the program lead to reduced drop-out rates?

Example Measures and Sources

(1) Average number of students attending each Personal Growth classes

(2) Number of Personal Growth classes Held

(3) Student knowledge of curriculum covered during Personal Growth classes

(4) Curriculum units actually delivered

(1) Average number of students attending Movie Night

(2) Number of Movie Nights held

Number of students referred to the program.

Characteristics of students referred to the program (do they meet eligibility requirements?).

Survey of students done after each curriculum session asking for specific feedback.

Survey of students done after each movie asking about the content of the movie and how they perceived it.

Characteristics of students attending the movie sessions, and whether or not they are in the PG classes?

Social network analysis of peer groups.

Survey scores on self-report. Hawkins & Catalano School Student Survey.

In-depth interviews with selected program participants asking them how the program helped them and how it could be improved.

Archival indicator data.

Analysis

Counts of events and classes

Average number attending

Quiz scores for curriculum knowledge (% correct).

Frequencies and counts of number of students referred and whether they match at least one eligibility requirement.

Summary of open-ended responses.

Review of responses to surveys to determine whether curriculum is having intended effects.

Count of the number of new peers students report having.

Pre-Post measures taken at referral to the program and at the end of the school year.

Summary of in-depth interviews highlighting key themes.

Percent change from last year on selected indicators.

In order to address the level of this risk or protective factor:

(goals)

We will do the following program activities

For these people and for this amount of time

(target population)

We expect that this activity will lead to changes in these factors, ________________ which in turn will lead to our program goal

(theory of change)

We will know these changes have occurred if:

(short term outcomes)

We will know we are reaching our goals if:

(long term impacts)

(1) Low Commitment to school & school bonding

(2) Bonding to people with healthy beliefs and clear standards

(3) Resistance Skills*

(4) Opportunities for community services

(1) Elder mentor program: Students will be matched with an older adult with whom they will spend a minimum of two 2-hour sessions for the duration of the school year and over the summer. Mentors will help students with homework, provide tutoring, do volunteer community services, and engage in other recreational activities.

(2) Students will also perform community service by making biweekly hourly visits to a nursing home to talk and read with institutionalized frail elderly.

(3) A series of social activities to facilitate interactions between teachers, parents, and youth will be held. Events will be held monthly.

(1) 6th graders at XYZ Elementary school who are identified as at-risk for school drop-out because of at least one of the following:

*Held back a grade

*Failing more than one class

*Three or more incidences of detention or suspension for behavioral problems

(2) The parents of these 6th graders will also be invited to participate in social events

Elder mentors will serve as positive role models for at-risk youth, and they will provide positive feedback and individualized attention to the youth. This attention and positive feedback will help to increase the youth’s self-esteem and self-worth. The youth may also learn about the benefits of a drug-free lifestyle from the elder mentor. Tutoring may directly address school performance.

Engaging in community service work with institutionalized frail elderly will boost the youth’s self-confidence and their social skills. These activities may also instill a sense of social responsibility.

Increasing youth’s self-esteem, self-confidence and social skills will help them to resist peer pressure to use ATOD.

[*Note: it seems that nothing in the program directly teaches or reinforces students’ resistance skills. A skill-building module could be introduced to address this risk factor. It is unlikely that the program as currently described will influence students’ resistance skills]

Youth show increased self-esteem, self-confidence, and positive social skills.

Youth should also report a positive relationship with at least one adult.

Youth should report a sense of community and positive attitudes about community service.

Increased positive communication between parents, youth, and teachers.

Reduce school drop-out rates

Reduce rates of ATOD use

Increase student attendance rates

Improve student’s school performance.

[Note: this program is also likely to have positive outcomes for the elderly mentors and institutionalized elders. Depending on the purpose of your evaluation, it might make sense to evaluate these outcomes as well]

Example Evaluation Questions

How many mentors are recruited?

How many students are paired with a mentor?

How many different activities do mentors do with students?

How much time is spent on homework and tutoring?

How many mentoring sessions does each student have?

How many students visit the nursing home?

How many social events are held?

How many parents, teachers, and student attend the events?

How many students are successfully referred to the program?

What are the characteristics of those referred? Do the meet legibility requirements, and if not, why?

Do mentors and students have a positive relationship?

Do students value the time spent with their mentors? Do they perceive it as helpful?

Are mentors clearly communicating a non-use message?

How do students feel about visiting the nursing home? Do they see this as valuable?

Do parents feel the social events help them to know their students’ teachers better?

Do mentored youth show increased self-esteem, social skills, and self confidence?

Do mentored youth feel more committed to school?

Do youth who visit nursing homes have more positive attitudes about community service?

Do parents who participate in social events communicate more frequently with teachers?

Do students participating in the program have reduced drop-out rates?

Does the program lead to reduced rates of ATOD use?

Does the program lead to increased student attendance rates?

Does the program lead to improved student school performance?

Example Measures and Sources

Number of students successfully paired with a mentor.

Number of hours of mentoring provided for each student.

Duration of mentoring (number of weeks).

Types of activities conducted during mentoring sessions, amount of homework, social, tutoring, etc.

Number of social events held.

Number of students, teachers, and parents attending social events.

Number of students referred to the program. Characteristics of students referred to the program (do they meet eligibility requirements?).

Surveys of youth including ratings of their feelings about their mentor and the mentorship program.

Focus groups with youth and mentors.

Surveys of students after visits to nursing homes.

Student’s self-report, Hawkins & Catalano School survey;

Increase in youth’s self-esteem scores, using the Rosenberg self-esteem scale

Increase in students’ sense of belonging to a community, using the "Sense of Community" scale.

Increase in reported number and opportunity for communication between parents, teachers, youth

Archival indicator data.

School survey data.

Analyses

Frequency and counts

Frequency of counts and % who match eligibility requirements

% of students reporting "positive" relationship with mentor

Frequency counts of which aspects of mentoring are most important.

Summary of major themes from focus groups.

% of students reporting that visits to nursing homes are valuable.

Pre-post scores of youth measured at referral and at the end of the summer.

Increase in self-reported communications between parents, youth, and teachers measured with brief phone survey at the end of the school year. % who report an increase.

Percent change from last year on selected indicators.

In order to address the level of this risk or protective factor:

(goals)

We will do the following program activities:

For these people and for this amount of time

(target population)

We expect that this activity will lead to changes in these factors, ________________ which in turn will lead to our program goal

(theory of change)

We will know these changes have occurred if:

(short term outcomes)

We will know we are reaching our long-term goals if:

(long term impacts)

(1) Friends who use

(2)

Favorable attitudes towards drugs

(3) Family bonding

(4) Healthy beliefs and clear standards

(5)

Community laws and norms.

There are 5 components to this program. They each focus on somewhat different sets of risk factors

1) School-Based Program

A social influence curriculum will be delivered by trained teachers, involving 13 lessons in year one, and a 5-lesson bolster curriculum in year 2.

Lessons take 45 minutes.

Additionally, students are given exercise to do with their parents as homework.

All 6 & 7th grade students in 3 district middle schools and their parents. Eighth grade students during Year One will be the comparison group for the study.

Currently there are approximately 100 students in each grade at each school, for a total of 600 students.

The curriculum focuses on resistance training, and developing skills to resist pressures to use drugs. Students who develop these skills will be less likely to use drugs.

Additionally, the curriculum provides information about the negative consequences of drug use and the positive aspects of a drug-free lifestyle. Students who are exposed to this information will be more likely to develop standards and norms of non-use, leading to reduced use.

Activities between parents and children focused on will improve the quality of relationships between parents and children and increase open communication about drug use and its consequences. This will increase the chances that students will adhere to a drug-free life style.

Students completing the classes show improved resistance skills, less favorable attitudes towards drug use, and report fewer friends who use ATOD.

Students should also report more positive relationships with their parents.

Reduced ATOD use among students.

Example Evaluation Questions

How many students participate in the sessions?

What is the average length of the sessions?

How many parents work with their children on homework?

Do all 6th and 7th graders participate in the program?

Which parents tend to participate? Are we risking the families most at-risk?

Do students enjoy the sessions?

Are sessions developmentally appropriate for this age group?

Is information presented credible? Do students believe that non-use is the norm?

Are activities successful in engaging both parents and children?

Do parents feel more comfortable talking with their children about drug use?

Do students participating in the program develop more resistance skills, social skills, and more favorable attitudes?

Do students develop friendships with non-using peers?

Do parents and children communicate more frequently?

Does the program reduce rates of ATOD among students?

Possible Methods, Measures, and Sources

Attendance records

Timesheets from classes

Parents survey (reports of working with children on homework

Attendance Records

Student records

Focus group with students participating in the sessions.

Survey of students participating in sessions.

Survey of parents.

Survey of students done to assess social and resistance skills, pre-post.

Interviews with parents and students

Archival data

Possible Analysis strategies

Frequency count

% of parents reporting doing homework with children

Comparison of attendance records and enrollment

% of parents participating whose students are "at-risk"

Synthesis and coding of focus group notes.

% of students reporting that information is credible and that non-use is a norm.

% of parents reporting feeling comfortable talking with students about ATOD use

Pre-Post differences on survey scales.

Synthesis and coding of interview.

Percent change since last year in archival indicators.

In order to address the level of this risk or protective factor:

(goals)

We will do the following program activities:

For these people and for this amount of time

(target population)

We expect that this activity will lead to changes in these factors, ________________ which in turn will lead to our program goal

(theory of change)

We will know these changes have occurred if:

(short term outcomes)

We will know we are reaching our long-term goals if:

(long term impacts)

Family bonding

Healthy beliefs and clear standards

Community laws and norms

(2) Parent Program

The parent program fosters parent involvement in school and community-based prevention activities. Parents are:

*encouraged to participate with their children on homework activities from Component #1

*Encouraged to attend a school-based parent organization that sponsors alternative student activities, does fund raising, and promotes school and community drug-free policy [note: you might want to evaluate the activities of the parent organization separately, for example, tracking the number and attendance at sponsored events]

*Parents in the parent organization will be provided with 2 2-hour training sessions

Parents of children participating in component #1. Estimate: 50% of parents will participate in homework.

Parents who participate in school-based parent group.

Estimate: 25-35 parents will participate in parent organization in each school.

Parents who participate in helping their children with their homework will develop more positive relationships with their children. Children who have more positive relationships with their parents are at lower risk for ATOD use.

Parents who participate in the parent organization will implement activities that promote positive aspects of a drug-free lifestyle, and which support community norms for non-use. Note: depending on what activities the parent groups plan, there might be different "theories of change" that could be included here.

Parents will learn more about how to positively communicate with their child about ATOD, and will be better able to set clear standards for their children. This communication and clear standards lead to children being less likely to use.

Parents report more positive relationships with their children.

Parents report more negative attitudes about student ATOD use, and more positive attitudes about drug-free lifestyles.

Parents report increased knowledge of the negative consequences of substance abuse.

Parents report having better communication skills and feel more comfortable talking with their kids about ATOD.

Reduced rates of substance abuse in the community.

Reduced student ATOD use.

Example Evaluation Questions

How many activities does the parent organization sponsor?

What kinds of activities do they sponsor?

How many activities are specific to ATOD prevention?

What percentage of parents participate in the parent organization? In the homework?

How many participate in trainings?

Do students bring homework home?

Do parents work on homework with students?

How do parents make decisions about what activities to plan? To what extent are activities focused on ATOD prevention?

Do parents feel supported by other parents? Do they feel a sense of community with other parents?

Does doing homework together help communication?

Do parents gain knowledge about the consequences of ATOD use?

Do parents increase in communication skills?

Do parents feel more comfortable talking with their children about ATOD use?

Is the number of students using ATOD reduced?

Possible Measures and Sources

Attendance Logs

Minutes from parent group meetings

Activity descriptions

Attendance Log

Parent survey

Student survey

Parent Survey

Focus group with parents in parent organization

Parent Survey of knowledge of ATOD use

Parent report of level of comfort talking with children about ATOD use

Archival Data

Example Analyses

Count of number of persons participating in each activity

Count of number of parents participating in parent organization

Qualitative analysis and coding of types of activities

Count of number of persons participating in each activity.

% of parents in program reporting that they do homework with children

% of students reporting bringing homework home

% of parents reporting that they do homework with students

Categorization and coding of focus group notes

Pre-post differences on students’ skill scores

% of parents reporting an increase in comfort level in speaking to their children about ATOD

% change since last year.

In order to address the level of this risk or protective factor:

(goals)

We will do the following program activities:

For these people and for this amount of time

(target population)

We expect that this activity will lead to changes in these factors, ________________ which in turn will lead to our program goal

(theory of change)

We will know these changes have occurred if:

(short term outcomes)

We will know we are reaching our long-term goals if:

(long term impacts)

Healthy Beliefs and Standards

Community Laws and Norms

Availability of Drugs (tobacco)

(4) Community Organization & (5) Health Policy Changes

A local community coalition of volunteers and leaders from all sectors of the community will oversee the implementation of Project Star. This coalition will be staffed by a half-time coordinator who will organize monthly coalition meetings. The coalition will provide volunteer staffing and other support to the other components of Project Star

The activities of the coalition include monthly meetings to facilitate communication between community sectors, information and resource sharing, and developing additional community-based activities to support ATOD prevention. During year one, the coalition will focus on two activities:

(1) A Drug Awareness March

(2) Establishing and monitoring "smoke free zones" around the middle schools. To do this we will:

*lobby the schools to establish tobacco free zones and post signs indicating "no smoking" in and around school property

*Local business will be encouraged to post signs in and around their property

*local vendors will be encouraged to post signs indicating the proof of age is required for sales of cigarettes.

Participants in the coalition will include representatives from the following groups: Parents, youth, school administrators, school counselors, County Health Division, local mental health providers, policy department, faith community, and local business. We hope to have at least 25-35 people participating in the coalition.

The drug awareness march will involve all students and teachers at the 3 middle schools, plus additional community members. Estimate: 700 people.

All 3 middle schools, plus the key 6 businesses in the vicinity of these schools will be targeted for "tobacco free zones"

By involving a variety of different community sectors, the likelihood of changing community norms related to ATOD is increased. Changes in community norms are expected to lead to lower rates of use and abuse.

Information and resource dissemination will increase community residents’ knowledge of services available in the community for ATOD use and abuse. This knowledge will lead to more timely and frequent access to these community resources, and therefore help to reduce ATOD related problems.

The awareness march, although not in and of itself expected to influence risk factors, plays a role in publicizing the activities of Project Star and the Coalition, and helps to gain community support for our activities.

Instituting tobacco free zones both sets a clear standards that smoking is not an acceptable behavior (supporting a community norm for non-use), as well as putting into place barriers to availability. If students are not allowed to smoke around school, their use may be reduced. Making cigarettes more difficult for youth to obtain may decrease tobacco use.

We successfully involve the key community sectors in our coalition

Community residents report awareness of Project Star and support its activities.

Community residents are more aware of community resources for ATOD-related problems.

Community residents report changes in their attitudes about students ATOD use, such that smoking is seen as a more serious problem or youth.

Community business are less likely to sell tobacco to minors.

Reduced rates of ATOD use, especially smoking, around youth.

Reduced ATOD use among community residents

Reduced numbers of local business selling tobacco to minors.

Example Evaluation Questions

How many coalition meetings are held?

Is the drug march implemented?

How many vendors post signs?

How many volunteer to help with lobbying efforts?

How many signs are posted at schools?

How many people attend coalition meetings?

Does the coalition include at least one representative from each sector?

How many community residents participate in the march?

Do coalition members communicate effectively?

Are resources and information shared at coalition meetings?

Does the awareness march effectively publicize the coalition’s activities?

Are students aware of the tobacco-free zones?

Did vendors sell to minors before the campaign?

Do they feel it is more difficult to obtain cigarettes?

Do participants in the coalition report changes in their attitudes about student ATOD use?

Do community residents report changes in their attitudes about student ATOD use?

Do students find it more difficult to purchase cigarettes from local vendors?

Are vendors asking for identification from minors purchasing cigarettes?

Does the coalition have an effect on rates of community and student ATOD use?

Example Sources and Measures

Coalition meeting minutes

Attendance forms

Count of number of signs posted

Volunteer sign-in sheet

Attendance forms asking for professional affiliation

Observation of number of marchers

Survey of coalition members

Coalition meeting minutes

Community survey

Student survey

Community survey

"Key informants" who’ve attempted to buy cigarettes.

Archival data

Example Analyses

Count of number of meetings

Count of number of signs

Count of number of resources shared at each meeting

Frequency count, percent of members representing each sector

% of community residents who report knowledge of the march

% of community residents who are aware of the coalition

% of students who report awareness of the tobacco free zones

% of students reporting a change in the perceived difficulty of obtaining cigarettes

Pre-post difference scores in the perceived difficult of obtaining cigarettes

% of businesses reported by key informants as selling tobacco to minors; % change in this number pre and post sign campaign

Percent change from last year’s data.