Sequencing Effects in the Analysis of Complex Experiments in Business Research: Mechanisms, Biases, and Recommendations

Abstract

Experiments in business research became more complex over time, yielding complex sequences of stimuli and measurements. This raises the issue of sequence effects, where effects are found only in specific sequences of the experiment. One case in point is factorial surveys. Here, presenting the stimulus is followed by asking subjects to evaluate several vignettes presented in a certain sequence. The researcher is interested in the effect of the stimulus on responses to vignettes with certain features. As sequence and stimulus can be made uncorrelated by construction, holding the sequence constant or excluding the sequence from the analysis seems to be justified when researchers are only interested in effects of vignette features or the stimulus. In both cases, even if the sequence is relevant for the dependent variable, correlation between sequence and stimulus, the necessary condition for an omitted variable bias, is absent. The effect estimated for the stimulus should thus be unbiased. We show that even in the case where stimulus and sequence are uncorrelated or the sequence is held constant, an omitted variable bias occurs when the effect of the stimulus in a vignette is in its magnitude dependent on the sequence in which the vignettes were presented. Such an effect would be modeled by including a sequence‑stimulus‑interaction term and the omitted variable is this interaction term, which is, by construction, always correlated with each of the constitutive variables. A simulation is presented to illustrate the problem. Implications for experimental research are discussed.