![](/navimages/spacer.gif"){kind=spacer}
|
||||||||
|
|
|
|
||||||
|
|
|
||||||
|
|
Volume 7: No. 3, May 2010
ORIGINAL RESEARCH
|
||||||||||||||||||||||||||
| Box. RE-AIM Components and Operational Definitions | |
|---|---|
| RE-AIM Component | Definition |
| Individual participation rate (IPR) | The percentage of eligible members of the insurance agency who participated in the weight management program |
| Demographic representativeness (DR) | How well participants represented the population eligible for the program |
| Individual completion rate (ICR) | The percentage of participants who completed each of the phases of the weight management program |
| Outcomes (O) | Weight change experienced by completers of each phase of the weight management program |
| Differential impact (DI) | A measure of differential changes in weight experienced among phase I and II completers by sex and age |
| Setting participation rate (SPR) | The percentage of eligible cardiac rehabilitation centers, physical therapy clinics, wellness/fitness centers, and health clubs in the state that participated in the weight management program |
| Component implementation rate (CIR) | The percentage of program protocols implemented at each site |
| Setting continuation rate (SCR) | The percentage of program sites that have elected to continue conducting the program for more than 1 year |
Presenting results using the RE-AIM model first requires creating operational definitions for multiple components in each of the 5 RE-AIM dimensions (10) (Box). These components were used to calculate RE-AIM indices (Appendix). Index calculations used standardized effect size (18) values from various statistical procedures to measure multiple components or outcomes in each index. We used χ2 analysis to compare categorical variables and repeated measures analysis of variance (ANOVA) to assess the magnitude of outcome changes over time. In the indices, “positive” effects were reduced by differential or “negative” effects (eg, program attrition). This index calculation method has resulted in negative values in prior studies (13) for 2 reasons. First, in calculating index values, differential effects have been subtracted directly from outcome effects, overstating the “negative� program effect. To limit this potential distortion, we calculated a “proportion of positive effect not explained by differential effects� by first subtracting the differential (“negative�) effect size from 1 and then multiplying the difference by the “positive� effect (Appendix). Second, a negative index value may still result if the effect size is more than 1. We resolved this challenge by using effect size measures whose upper limit was 1.
Effect size measures (and interpretation) included the squared Cramer phi (φc2) for χ2 tests (small [φc2 ≈ .01], moderate [φc2 ≈ .09], or large [φc2 > .25]) and the squared curvilinear correlation coefficient (partial eta squared; η2) for ANOVA (small [η2 ≈ .01], moderate [η2 ≈ .06], or large [η2 > .14]) (18,19). All raw index scores ranging from 0 to 1 were multiplied by 100 for summary index values of 0 to 100. Missing data were excluded from analyses, which were conducted by using SPSS version 14.0.
The data for 1,952 participants from 31 sites were exported, and administrators from 27 of 31 active sites (87.1% response rate) completed online surveys. Responses from 4 new sites whose participants had not yet completed phase I of the program were excluded from analysis, resulting in 23 completed site administrator surveys. The distribution of most of the participants� measurement data was skewed, and data are presented as median (interquartile range) unless otherwise noted.
A total of 60,041 adult members were covered by the insurance agency (N. Henderson, health promotions director, written communication, January 2008). By using the state obesity (31%) and overweight (36%) prevalence rates (20), and national prevalence of overweight adults with at least 1 comorbid condition (66%) (21), we estimated that 32,878 insurance members may have been eligible for the program. At the time of data collection, 1,952 members had participated (5.9% individual participation rate) (Table 1).
Baseline physical measurements for men and women are presented in Table 2. A significantly larger percentage of participants than the eligible population were women (80.0% vs 54.1%; P < .001, φc2 = .073), and more of them were aged 45-54.9 years and fewer of them were aged 65 years or older (P < .001, φc2 = .113), yielding a reach value (R) of 5.4 (Table 1).
Phase I outcome changes are presented in Table 3. Of 1,647 participants who could have completed phase I (those who had started the program at least 12 weeks before data collection), 76.5% had done so. Fewer women (74.8%) than men (83.7%) completed phase I (P = .001, φc2 = .007), and completion rates tended to rise with age. Participants who completed phase I had significant weight loss (13 lb [6.5-21.4 lb]; P < .001, η2 = .592), and men lost significantly more weight than women (P < .001, η2 = .050). Differences in weight loss were also revealed among age groups (P = .003, η2= .014). These components produced an effectiveness value (E) of 43.8 and an individual-level impact ([R * E] / 100) of 2.4 (Table 1).
There were 31 active weight management program sites out of a total of 352 potential sites in West Virginia at the time of data collection, resulting in an adoption value (A) of 8.8 (Table 1). Site survey responses (N = 23) showed that sites implemented a mean of 12.8 (standard deviation [SD] 1.0) of 14 program components during phase I. All sites measured weight, waist circumference, body fat percentage, and blood pressure; tracked attendance; and provided initial and follow-up nutrition assessments with a registered dietitian, nutrition classes designed by the registered dietitian, and an individualized exercise prescription. The least frequently implemented component, by 15 sites (65%), was having participants maintain home exercise logs. Implementation survey data yielded an implementation value (I) of 91.4 and a setting-level impact ([A * I] / 100) of 8.0 (Table 1).
Phase II outcome changes are presented in Table 4. Of 762 participants who could have completed phase II (those who had started the program at least 1 year before data collection), 348 (45.7%) had done so (ICRPhII = .457) (Table 1). Excluding the small samples of participants aged 18-24.9 years (n = 2) and 65 years or older (n = 9), completion rates tended to increase with age and be higher in men in each age group. Participants who completed phase II achieved significant weight loss from baseline (15 lb [6.1-30.3 lb]; P < .001, η2 = .467), shedding 6.7% (2.7%-12.7%) of baseline body weight. Weight loss was similar among phase II completers of different age groups (P = .61, η2 = .011) and between sexes (P = .21, η2 = .005). These values combined to yield an individual maintenance value (MI) of 21.2 (Table 1).
Four of 18 sites (22%) that had been approved to accept participants at least 1 year before this study had stopped or had been disallowed to continue accepting participants, resulting in a setting maintenance value (MS) of 77.8 and a long-term maintenance value ([MI * MS] / 100) of 16.5 (Table 1).
We achieved our primary objective of using RE-AIM to evaluate a weight management program. We found moderate program effectiveness and high implementation, suggesting the program has been beneficial for participants and can be implemented in a variety of settings. We found low program reach and adoption, suggesting the program could be improved by recruiting new participants and sites. Recruitment may prove difficult, however, because participants must be highly motivated to enroll in the program, and sites are required to offer services by highly trained personnel often unavailable in rural areas of West Virginia. In the long term, site maintenance was high, but individual maintenance was fairly low, indicating the program is sustainable but the services of phase II may need to be revised to improve participant outcomes. The summary results suggest this weight management program has potential to be expanded for more translation and public health benefit and should be considered a viable model for other public and private insurers.
Individual short-term and long-term outcome changes are comparable with those of other behavioral programs and clinical trials. Short-term attrition from this weight management program (23.5%) was slightly higher than is commonly seen in behavioral programs of similar length (10%-15%), though the median weekly weight loss in this program is comparable (1.23 lb vs 1.1 lb) (22). This program also compares favorably with clinical trials of similar length, which average 85%-95% completion rates and approximately 1 lb of weekly weight loss (1,5).
Long-term individual results also compare favorably with other behavioral programs and clinical trials. Participants who completed phase II (n = 348) lost a mean of 20.9 (SD, 22.3) lb from baseline, with some recidivism. Slightly more than half of phase II completers (51.5%) maintained phase I weight loss or continued losing weight in phase II. In comparison, approximately 60% to 70% of weight loss is maintained for a year after treatment in other short-term behavioral interventions (22). One-year results indicate more average weight losses but lower completion rates than randomized control trials of similar length (1,5).
We also achieved the secondary objective of critiquing the RE-AIM model and revised index calculations. The strength of the RE-AIM model is its ability to quantify for decision makers a program’s strengths and weaknesses. Comparison with other health promotion program evaluations that used RE-AIM is limited at this point in the model’s refinement because no 2 studies have used the same index calculation methods. We believe this study advances the RE-AIM model by 1) providing methods for assessing long-term maintenance at the individual and site levels and 2) addressing 2 methodologic concerns with existing index calculation methods (ie, negative index values and effect sizes with varying maximum values). The revised methods in this study produced positive R and MI values, whereas previously used methods (10) would have yielded negative values.
The study is limited in a number of ways. Multiple sources of measurement error may have affected the data, including 1) lack of standardized procedures and instruments for measuring health outcomes, 2) missing outcome data, and 3) social desirability of sites when entering participant data and completing survey items. Potential error was addressed in multiple ways. Trained exercise professionals took measurements using accepted professional standards of practice, the insurance agency periodically audited site data, we contacted sites to collect missing data, and all survey recruiting material stressed the informative (not punitive) nature of the study and independence of the investigators from the insurance agency. Measurement error would be more important in a small clinical trial assessing an intervention’s efficacy than in this study with its large sample and focus on standardized effect sizes.
Numerous questions remain to be answered before the public health effect of this and other weight management programs can be better understood. This program had a significant positive effect on participants and was sustainable, but needs to be expanded. The RE-AIM model provided a framework through which the translation of this program could be evaluated and presented to public health decision makers. In fact, the summary data of this project were presented to the insurance agency and used as evidence for changing the program benefit to address the low reach, adoption, and long-term individual maintenance. We encourage continued study of RE-AIM index calculations and application of the model to evaluate obesity treatment programs. It may be beneficial to add qualitative process data to these outcomes to work toward developing a set of best practices in behavioral weight management (11,12). Additionally, because insurance agencies often provide benefits for multiple weight management modalities (eg, behavioral, surgical, pharmacologic), RE-AIM methods should be used to evaluate multiple modalities concurrently to allow for side-by-side comparison and facilitate decision making about resource allocation. These evaluations will need to link program costs, insurance claims, individual outcomes, and future cost savings. Such analyses may affect the benefit and incentive structure of this insurer and others to improve the translation of knowledge regarding clinical obesity treatment into innovative programming that can be widely implemented.
This research was funded by the State of West Virginia Public Employees Insurance Agency.
Corresponding Author: Christiaan G. Abildso, PhD, MPH, West Virginia University College of Physical Activity and Sport Sciences, PO Box 6116, Morgantown, WV 26506-6116. Telephone: 304-293-0860. E-mail: [email protected].
Author Affiliations: Sam J. Zizzi, Bill Reger-Nash, West Virginia University, Morgantown, West Virginia.
|
|
|
|
|
|
|
|
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
| This page last reviewed March 30, 2012
|
|
