Using a variation of the cohort control design to evaluate large-scale, long-term, complex professional development programs
Date
2009
Journal Title
Journal ISSN
Volume Title
Abstract
The purpose of this study was to introduce a variation on the posttest-only cohort control design and answer questions concerning both methodological credibility and practical utility of employing the variation design in evaluations of large-scale, complex professional development programs. The original design and variation design, which adds a pretest measure for prior student performance, were compared theoretically and practically using data from the RM-MSMSP program to compare the advantages and disadvantages of the two evaluation designs. Two separate 2 x 2 ANOVA analyses were used to compare the designs. Findings indicated that, as expected, there were differences in the outcomes using the two designs. While the outcomes were not consistently different, they could plausibly be explained. Because the findings of the variation design were supported by similar findings in the literature, credence was given to the variation design. Given the added control of the variation design, evaluations using the variation design could control for selection bias where those using the original design could not. Realistically, however, the choice of evaluation design is one of trade-offs, because the addition of controls through gain scores, as is the case for the variation design, also comes with some disadvantages. If certain conditions for the data are met and the outcome measure is calibrated, the variation design would be a good choice for a professional development evaluation. If not, the original design would also be perfectly reasonable. From a practical standpoint, the variation design is no less practical to employ than the original cohort control design if certain program conditions about data collection and availability are met. In addition, the outcome measure in the form of a gain score is similar to value-added evaluation designs that are politically popular due to a focus on student growth. This coupled with the methodological advantages of the variation design make it a useful evaluation design for large-scale, long-term, complex professional development programs wishing to investigate the effects of professional development on student achievement.
Description
Rights Access
Subject
cohort control design
complex PD evaluation
design comparison
evaluation design
professional development
quantitative evaluation
educational tests and measurements
school administration
curriculum development
educational administration