Testing the List Order Response Effect Among Respondents With Cognitive Sophistication: Experimental Evidence in Management Information Systems Research

Abstract

Questionnaires constitute a valuable data-collection tool in Management Information Systems (MIS) research. However, MIS researchers have identified various biases in the design and implementation of questionnaires. This paper focuses on the bias resulting from the order of items in the answer choices, called list order bias. Such bias is described through a framework of cognitive theories, including the cognitive elaboration model, memory limitation hypothesis, and satisficing theory. Previous literature has proved that satisficing theory is superior in explaining list order bias; therefore, such theory is adopted for this study. Satisficing  theory posits that respondents provide a satisfactory rather than an optimal answer when a survey question requires cognitive effort. Previous research has shown that satisficing is triggered by respondents' cognitive abilities to complete the questionnaire and, therefore, it is predominant among less educated respondents. However, the extent to which satisficing behaviors could occur, even among respondents with higher education and cognitive abilities, still needs to be ascertained. This is particularly important for MIS studies that investigate information systems' adoption at the organizational level because they rely mostly on respondents who are information technology (IT) managers. Therefore, this study adopts the satisficing theory to examine the list order response effect among cognitively sophisticated respondents in the MIS field. The authors selected and manipulated a question from the Society for Information Management's (SIM) IT Trends Study web-based questionnaire to conduct such an analysis. The SIM IT Trends Study survey questions offer a lengthy list of answer options to SIM members who are IT managers inside organizations that operate in various business sectors. The authors created two types of the same list question: one provided the list options in alphabetical order and the other provided the list answers in reverse-alphabetical order. The findings show statistically significant empirical evidence for list order bias by revealing that, despite their cognitive sophistication, respondents were more likely to choose the first available answer, especially in the case of reverse-alphabetical order. In light of these findings, the authors propose remedies to decrease the satisficing behaviors of such respondents. In particular, researchers could break questions with long lists into several questions with short lists and then combine those responses into the answer selection list of a final question. Researchers could also provide the answer selection lists to half of the sample alphabetically and the other half in reverse order and then combine the two subsamples into the final possible responses. Alternatively, researchers could use "trigger" or "priming" statements before displaying the question and its answer selection list to reduce the questionnaire's difficulty. In summary, this study addresses the list order response bias among respondents with cognitive sophistication in MIS research, explains why this bias occurs by employing satisficing theory, and provides remedies for reducing the relevant occurrence. Hence, this manuscript contributes to MIS research by providing insights to improve the quality of questionnaires by minimizing satisficing behaviors that lead to list order bias, and it makes MIS practitioners aware of the possible influence of question design when they respond to questionnaires.