Research in Translation
ACT for Youth: Measuring Positive Youth Development
March 21, 2017
A myriad of evidence-based programs exist to help young people develop positive life skills and avoid risk behaviors. Evaluations of program success are frequently focused on whether programs directly reduce negative outcomes for youth. An alternative approach can be to measure Positive Youth Development (PYD) outcomes resulting from a particular program. Program evaluation with the PYD approach places the focus on positive, healthy outcomes for youth, rather than a focus on reduction of negative outcomes.
In 2015, the New Jersey Department of Health (NJ DOH) asked the ACT for Youth Center of Excellence to provide program evaluation of Wyman’s Teen Outreach Program® (TOP®). TOP® is an evidence-based program that offers young people community-based, service-learning opportunities in order to build on their strengths and reduce risk behaviors. The NJ DOH facilitates the implementation of TOP® and has traditionally measured its success using Wyman’s own pre- and post-surveys. While these surveys provide useful data, the NJ DOH wanted to augment the evaluation with a PYD approach. The ACT for Youth team, including Amanda Purington, Christine Heib, and Dora Welker, was contracted to identify specific PYD measures that could be used in conjunction with the existing evaluation approach, and to develop a survey tool for measuring progress in PYD outcomes.
To develop the new surveys, ACT for Youth used the Positive Youth Development Inventory – Very Short Form.These measures are based on the Lerner and Lerner “Five Cs” model of PYD, which encompasses the “Cs” of youth development: Competence, Confidence, Connection, Caring, and Character. Each of these “Cs” are measured using subscales that assess different aspects of youths’ lives. The PYD surveys created by ACT for Youth incorporate these subscales to get a more comprehensive picture of youths’ lives and the impacts of the TOP® program.
While funding for the evaluation came from the NJ DOH, implementation was done through the Central New Jersey Family Health Consortium, Inc. (CJFHC). Youth completed the PYD pre- and post-surveys at five separate programming locations where TOP® was implemented during the 2015-2016 school year. Additional information such as demographics, living situation, year in school, and school attendance were also collected, enabling ACT for Youth to conduct a more comprehensive analysis of program effects.
After looking at results of the PYD pre-surveys, ACT for Youth found high baseline scores for all five of the Cs. This was a very positive finding -- however, it did create a challenge for evaluation, since with such high initial scores there was not much room for further improvement. While the evaluation did not detect much improvement, and saw some decreases, overall these high scores were largely maintained at post-survey.
To better understand differences among the results, the ACT for Youth team also analyzed pre- to post-survey differences in relation to a variety of grouping variables, such as attendance and “baseline risk,” which was measured on the Wyman pre-survey. Examples of risk factors in this survey include: “failed a course,” “been suspended,” “been pregnant/caused a pregnancy,” and more. ACT for Youth grouped and coded these risk factors into “academic baseline risk” and “sexual health baseline risk” and used these baseline risk levels to further assess TOP® program efficacy, taking into consideration the diverse backgrounds of youth participants and their very different starting points at the beginning of TOP® programming. The goal of this analysis was to see if TOP® is more effective among certain distinct populations. Interestingly, some of the findings from this analysis suggest that TOP® implementation may have some of its most positive effects on the most vulnerable youth.
Creating and using the PYD surveys allowed the evaluation team to assess positive youth development, alongside traditionally measured negative risk factors, for a more comprehensive and optimistic evaluation of TOP® in New Jersey. While it was unexpected that a number of items moved in the undesired direction from pre- to post-survey, several reasons might explain this. First, as mentioned, with highly positive baseline scores it is difficult to achieve significantly higher results at post. Additionally, it is possible that implementation of the TOP® program mitigates what would otherwise be even greater declines in certain areas. In the future, this hypothesis could be explored by having youth not involved in TOP® programming also complete pre- and post-surveys as a way of assessing a control group.