Building robust statistical models for categorical outcome variables in educational research

Abstract

This thesis had a primary aim: to build robust statistical models with categorical outcome variables for intervention evaluation in educational research. It is critical for policy and equity reasons to know just how effective a school intervention is and under what conditions. Addressing this aim meant solving two challenges in value-added models in educational intervention research in New Zealand: (a) the lack of uniform valued student outcome and (b) the complexity of the data structure. The first practical challenge leads to the first research question of this thesis: how to choose between valued student outcomes for value-added models to measure the effectiveness of school interventions? Specifically, how to optimise reliability when fitting categorical outcome variables in a continuous outcome model and how to reliably use dichotomised continuous outcome variables in a binary outcome model? The second practical challenge leads to the second research question of this thesis: how to specify the model to achieve a desirable balance in the trade-off between model complexity and model validity? Specifically, how many underlying random effects should be fitted in a multilevel model with categorical out comes in order to reliably and effectively determine individual-and group-specific effects. Answers to these questions will be illustrated by the use of two case studies: one from a sustainability study based in two clusters of primary schools; and another from a large-scale, longitudinal research and development programme based in secondary schools. The findings from the primary school case study showed that with respect to the first research question: (a) categorical outcome variables can be reliably fitted in a continuous model as long as the underlying distribution is asymptotically normal (such as stanine scores in standardised tests) and (b) dichotomisation of continuous outcome measures results in a more conservative but more direct estimation of school intervention effects. The findings from the secondary school case study showed that with respect to the second research question: (a) model reliability and accuracy are improved when additional random effects are fitted and (b) the size of model improvement (in terms of reliability and accuracy)varies between random effects where improvements are mostly contributed by the inclusion of the lower-level random effects.