By: David L. Wyrick, PhD , Jeffrey J. Milroy, DrPH, Institute to Promote Athlete Health & Wellness, University of North Carolina at Greensboro; Muhsin M. Orsini, EdD, Prevention Strategies, LLC
Most NCAA member institutions implement a variety of programs to promote the health and well-being of their student-athletes. Using substance abuse prevention as an example, common forms of programming include implementing strong policies, evidence-based educational programs and rigorous drug testing procedures. Individuals responsible for developing programs to prevent substance abuse typically employ various evidence-based strategies that are combined into a single program package. An example of a program package is an educational initiative designed to target three evidence-based strategies: social norms, goal setting and decision making. Each of these strategies can be considered a program component and these program components are packaged together to create a substance abuse prevention program.
The health and well-being of student-athletes is the primary focus of all athletics departments. However, we need to ask ourselves if we are taking the necessary steps to help ensure that the programming intended to promote the health and wellness of student-athletes are effective. In short, it is essential that ongoing evaluation of health promoting policies, programs, and procedures is occurring. In too many cases, evaluation is only considered at the point of determining the impact of a program. Impact evaluation is a summative evaluation strategy where the primary purpose is to make a judgment on the overall effectiveness of a program on any number of outcomes of interest (e.g., substance use and/or negative consequences). In this case, findings (positive or negative) are attributed to the summative effect of all program components.
While evaluating the effectiveness of programs is important, there are also evaluation methods that can be implemented in advance to better inform program development and enhancement. These evaluation methods are traditionally considered to be formative evaluation. Formative evaluation provides the opportunity to collect and use multiple sources of data to improve health promotion programs. For example, process evaluation methods can be used to answer questions about how the program was implemented, the number of participants served, dropout rates and how participants experienced the program. However, all too often, program developers are short sighted and do not employ innovative formative evaluation methods that can greatly increase the likelihood of developing a program that can create meaningful behavioral impact. Ideally, athletics departments will find value and will commit the resources necessary to making both formative and summative evaluation standard practice to ensure that student-athletes are exposed to programming that is truly the best practice for promoting student-athlete health and wellness.
The purpose of this article is to underscore the importance of formative evaluation for the purpose of informing program development and improvement. Although formative evaluation is critical for understanding how various components of an intervention function, most formative evaluation methods are limited in that they do not provide sufficient answers to questions about the individual contributions of program components (e.g., whether each component is effective or whether all components are needed). This limitation is primarily due to the fact that program components are typically studied within the program package.
Fortunately, there is an innovative approach to program evaluation that provides insight about the individual performance of program components. The Multiphase Optimization Strategy (MOST) is a framework developed by Dr. Linda Collins, Director of the Methodology Center at Penn State University (for a thorough description of MOST, visit http://methodology.psu.edu/ra/most). MOST is used for program development and improvement that is focused specifically on optimizing individual program components. Optimization is a process for making the most efficient use of available resources in order to achieve clearly stated program criteria. It is important to note that the optimization process operates with the understanding that further program improvements can always be made. MOST consists of three distinct phases with the first two focused on formative evaluation and the third focused on evaluating the impact of a program:
The concept of optimizing program components to produce the most impactful program package possible with available resources and consideration of contextual restraints should be particularly appealing for those athletic departments interested in getting the most “bang for their buck.”
The Institute to Promote Athlete Health & Wellness (IPAHW) located at the University of North Carolina at Greensboro (http://athletewellness.uncg.edu) is employing MOST to optimize myPlaybook, an evidence-based online alcohol and other drug (AOD) prevention program aimed at reducing AOD use among NCAA student-athletes. The goal is to optimize each myPlaybook component through an iterative application of MOST. The iterative process is designed with the goal that each subsequent revision of each program component demonstrates that it is incrementally better than the preceding one. The optimized myPlaybook components will then be packaged together to form a program capable of making a meaningful impact on student-athlete substance abuse. More than 60 NCAA member institutions and over 8,000 student-athletes have participated in the project thus far. To learn more about the optimization of myPlaybook or inquire about participating in the 2014 spring semester phase of the study, contact Dr. David Wyrick.
When programs are optimized, a clear bar is set for effectiveness, efficiency and economy. Ultimately, this process promotes innovation in the development of programs designed to promote the health and well-being of student-athletes. MOST represents a new pathway for optimizing programs. Whereas the authors view MOST as the gold standard for intervention development and optimization, we do not expect you to use MOST in every instance of developing and evaluating health promotion programs. However, MOST exemplifies the importance of considering programs as an assembly of multiple components which must be acknowledged when make improvements to existing programming.
The Institute to Promote Athlete Health & Wellness at the University of North Carolina at Greensboro (IPAHW) is a center of excellence in health promotion for athletes of all ages and levels of competition. The Institute’s vision is to improve the health and wellness of athletes through the translation of prevention research to effective policies and practices. Areas of focus include:
The IPAHW mission is to work collaboratively with a variety of organizations to provide educational resources and support through evidence-based programming, training, and research/evaluation to promote athlete health and wellness thereby fostering lifelong athlete development. IPAHW plans to accomplish this mission by providing valuable technical and research/evaluation support for organizations that serve athletes.Last Updated: Oct 10, 2013