Abstract:
This study responds to calls from aviation training professionals for standardised measures of language ability to assess cadet pilots from non-English-speaking backgrounds who are about to enter practical flight training programmes conducted in English. More specifically, the project involved the development and validation of the Overseas Flight Training Preparation Test (OFTPT), which is aligned with the International Civil Aviation Organisation’s (ICAO) Language Proficiency Rating scale (LPRS) but contextualised to the specific linguistic needs of ab initio pilots. The test is intended to have a diagnostic function, with the overall goal of classifying candidates into three levels of ‘readiness’: Ready, Minimally Ready and Not Ready.
Test development spanned four phases. Firstly, linguistic needs were investigated by interviewing twelve subject matter experts (SMEs) and surveying 56 ab initio pilots. This investigation was used in Phase 2 to design and construct two forms of the OFTPT, which includes listening, reading, and speaking components and is administered online. Two validation frameworks were used: Bachman and Palmer’s Assessment Use Argument (2010) and O’Sullivan and Weir’s (2011) sociocognitive framework. In Phase 3, the tests were trialled with 98 Japanese and Chinese ab initio pilots. In Phase 4, nineteen SMEs participated in three interdependent stages to set standards. Indigenous assessment criteria were elicited to establish threshold levels of performance, which were captured descriptively within performance level descriptors and then used in an Ebel standard setting procedure to validate test content and determine cut scores. These were inferentially linked to the ICAO LPRS.
The rating procedures divided test-takers into high, mid, and low bands of achievement with acceptable reliability. In terms of the ICAO scale levels, Ready test takers achieved in the high range of Level 3, Minimally Ready test takers achieved within the low to high range of Level 3 and those scoring in the lower band of Level 3 were Not Ready. These findings suggest that the ICAO scale is not finely grained enough to distinguish levels of linguistic readiness among ab initio pilots, nor does it adequately reflect the knowledge, skills and abilities valued by SMEs within this domain.