Using hierarchical linear modeling to measure school effects on the Colorado Student Assessment Program

Abstract

The purpose of this dissertation was to contribute a theoretically and empirically sound approach for analyzing and interpreting results from the Colorado Student Assessment Program (CSAP). This study was grounded in school effectiveness research, as the primary objective was to isolate the impact of school practice by controlling for student characteristics and school context. Furthermore, this investigation was designed to build on contemporary studies that have employed promising statistical models to analyze high-stakes tests. Thus, secondary databases were analyzed using hierarchical linear modeling (HLM) to identify the most consistent and powerful predictors of student achievement on the CSAP fourth-grade reading test. A value-added analysis also was implemented to estimate school effects and to compare school performance on this standards-based statewide assessment. The significance of this research is that interpretations of high-stakes testing (HST) can be made fair and accurate for all educational stakeholders. The two-level HLM analysis generated a predictive model based on the specification of significant student- and school-level variables. Specifically, prior achievement, special education status, socioeconomic status (SES), ethnicity, and continuous enrollment accounted for 64% of the within-school variance in student achievement. School SES and teacher experience accounted for 77% of the between school variance. The analysis of school effects estimates produced the most interesting finding, as performance differences and school rankings yielded contradictory results. For example, schools that outperformed expectations tended to have smaller value-added residuals than did schools that scored below their predicted value. Furthermore, there was a differential effect of school practice on students within a school based on prior achievement. The main implication of this study is that school accountability systems will be inequitable unless they control for the effect of educational inputs on HST results. The primary recommendation is that researchers, educators, and policymakers should utilize appropriate quantitative and qualitative methods to better understand and explain the complex dynamic between school practice and student achievement.