Abstract:
One of the essential properties of a valid research instrument is to be invariant for groups with different backgrounds (e.g., age, sex and ethnicity). There is a growth in the use of DIF strategies in social psychology research to test invariance in different rating scales attributable to participants' background characteristics. This study examined DIF of the Students' Conception of Assessment (SCoA-VI) instrument using the Pure Anchor MIMIC (M-PA) model. Motivation to reading and mathematics subjects was used as a grouping variable in the DIF analysis. Participants (N = 799) were year 9 and year 10 high school students in New Zealand. The grouping variable (motivation) was found to be domain specific consistent with the assumptions of self-regulated learning theory. The DIF analysis results were substantially consistent with self-regulated learning, specifically the dual model of motivation. When the performance of the MPA method of DIF detection was compared with the standard MIMIC model, the M-PA rendered improved model-fit with the relatively small samples used in the current study (N < 200 per group). This finding suggested that the M-PA can be a better choice for use when the reference and the focal groups have small sample sizes (e.g., N = 180). Very few items of the SCoA-VI were identified with potential DIF, and this was likely attributable to Type I error of over-identification of DIF associated with the lack of model fit. The pattern of DIF differed across subjects, suggesting that students' conceptions of assessment are subject-specific. The poor fit of the DIF models suggested that the sample sizes used in the current study are not sufficient for use with the M-PA procedure. Yet, interactional effects between the sample size and other model properties (e.g., factor loadings and number of factor indicators) appeared to influence the results of the analysis. Limitations of the current study, as well as implications for research and educators are discussed.
