Quantitative analysis selection in education: Potential impacts on researchers' conclusions

Abstract

This thesis utilises literacy data from a large professional development project in New Zealand schools to investigate the implications of researcher decisions about approaches to quantitative analysis. Specifically, the main research question posed is the extent to which the selection of particular analyses leads to incomplete or even erroneous conclusions when using real-world data. There are various publications advocating for particular methodologies based on this notion, but many of these use simulated data or extreme cases specifically chosen to demonstrate the advantage of the particular method being advocated. This use of simulated and manipulated datasets may explain why many researchers persist with analyses that are argued to be less than ideal. In this thesis, the implications of analysis choice are investigated using increasingly complex analyses, including effect sizes, single-level regression and multi-level models, with the results and conclusions made from each set of analyses collated and compared. A secondary aim was to evaluate how effectively the professional development project raised student achievement in literacy, and whether these shifts contributed to more equal outcomes for subgroups of students. Much of the previous research where student achievement is linked to professional development has been inconclusive. This inconclusiveness arguably increases the burden of evidence when examining the effect of professional development on student outcomes, so the extensive analyses utilised to investigate the primary thesis aim are especially useful in this regard, and support an in-depth examination of this secondary aim. The results of the investigations undertaken throughout this thesis showed that the professional development typically resulted in acceleration of progress rates (especially in writing) for all priority subgroups; including students in low socioeconomic catchment areas, and those of minority ethnicities. Choice of analysis was found to be of comparatively minimal importance when considering main effects, but secondary effects were susceptible to sometimes considerable differences in conclusions depending on the method of analysis used. These differences were of sufficient magnitude that policy decisions would likely differ depending on the type of analysis used to infer conclusions.