Principles and Techniques for Creating and Validating Computer-Adaptive Surveys (CAS)

Abstract

Diagnostic theories are fundamental to IS practice and are represented in trees. One tool for representing diagnostic theories is Computer-Adaptive Surveys. Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the responses of previous questions asked. Their principal advantage is they allow the survey developer to input a large number of potential causes. Respondents then roll down through the causes to identify the one or few significant causes impacting a correlation. As an example, consider a scenario where a café owner wants to know not only what aspect of customer service he could improve on, but also how he can improve. With traditional survey techniques, he would be forced to give customers a very long survey to identify the salient issues for improvement. In long surveys, respondents will suffer from a fatigue effect, and not answer questions properly. As CAS items have certain characteristics, such as being multi-dimensional and containing constructs with parent-child relationships, traditional methods for assessing construct and conclusion validity are not suitable. For this thesis, I have developed techniques and principles for creating and validating CAS which is applied to a CAS on café satisfaction. First, I created a variant q-sorting methodology for assessing the construct validity of the CAS tree. In that method, tree hierarchies that independent raters develop are transformed into a quantitative form, and that quantitative form is tested to determine the inter-rater reliability of the individual branches in the tree. The trees are then successively transformed to test if they branch in the same way. Second, the inter-rater reliability of the trees of raters need to be assessed. However, existing ways of measuring inter-rater reliability such as the use of Cronbach’s Alpha or the traditional way of using Cohen’s Kappa don’t work for CAS. I developed a way to measure inter-rater reliability in CAS using 3 measures of association, Goodman and Kruskal’s Lambda, Cohen’s Kappa, and Goodman and Kruskal’s Gamma. Third, I assessed the conclusion validity of a café satisfaction CAS using two studies. The first study compared a café satisfaction CAS with online customer reviews and the second study compared it to traditional survey of the same item bank.