Sources of Sample Bias in PISA: Selectivity and Effort Differences between Shanghai and New Zealand

Abstract

The Programme for International Student Assessment (PISA) is arguably one of the most influential international large-scale assessments (ILSAs) and Shanghai in 2009 and 2012 outperformed the world in all three test subjects. However, the legitimacy of such a comparison is questionable if sample bias exists among the jurisdictions. Two possible biases are evaluated in this thesis by comparing Shanghai (#1) with New Zealand (in the middle): the similarity of tested populations and the degree of effort in test-taking when a country’s reputation is at stake. In Study One, propensity score matching using demographic variables to control for background differences between the two samples reduced the performance gap by up to 57%. Matching based on performance scores found that the group of New Zealand students with similar performance to their Shanghai counterparts were from higher socio-economic backgrounds and somewhat more from urban neighbourhoods. Study Two examined Shanghai and New Zealand students’ test-taking effort and motivation in a between-subjects experiment with vignette instructions related to the test having one of three different consequences (i.e., none, country, or personal). In both jurisdictions, students indicated less motivation and effort when the country’s reputation was at stake but the gap to effort when personal consequences were at stake was much larger in New Zealand. This suggests Shanghai students are biased toward high test-taking motivation for ILSAs. Together, the two studies indicated that the large performance gap between Shanghai and New Zealand is likely to be a function of sample demographic differences and jurisdictional motivational differences in the context of the PISA ILSAs.