Using a variation of the cohort control design to evaluate large-scale, long-term, complex professional development programs - Sample McMeeking, Laura, author; Cobb, R. Brian, advisor; Kennedy, Paul, advisor (Colorado State University. Libraries, 2009)
The purpose of this study was to introduce a variation on the posttest-only cohort control design and answer questions concerning both methodological credibility and practical utility of employing the variation design in evaluations of large-scale, complex professional development programs. The original design and variation design, which adds a pretest measure for prior student performance, were compared theoretically and practically using data from the RM-MSMSP program to compare the advantages and disadvantages of the two evaluation designs. Two separate 2 x 2 ANOVA analyses were used to compare the designs. Findings indicated that, as expected, there were differences in the outcomes using the two designs. While the outcomes were not consistently different, they could plausibly be explained. Because the findings of the variation design were supported by similar findings in the literature, credence was given to the variation design. Given the added control of the variation design, evaluations using the variation design could control for selection bias where those using the original design could not. Realistically, however, the choice of evaluation design is one of trade-offs, because the addition of controls through gain scores, as is the case for the variation design, also comes with some disadvantages. If certain conditions for the data are met and the outcome measure is calibrated, the variation design would be a good choice for a professional development evaluation. If not, the original design would also be perfectly reasonable. From a practical standpoint, the variation design is no less practical to employ than the original cohort control design if certain program conditions about data collection and availability are met. In addition, the outcome measure in the form of a gain score is similar to value-added evaluation designs that are politically popular due to a focus on student growth. This coupled with the methodological advantages of the variation design make it a useful evaluation design for large-scale, long-term, complex professional development programs wishing to investigate the effects of professional development on student achievement.
Keywords: cohort control design; complex PD evaluation; design comparison; evaluation design; professional development; quantitative evaluation; educational tests and measurements; school administration; curriculum development; educational administration