Participant Carelessness: Is It a Substantial Problem With Survey Data?

Abstract

For decades, participant carelessness has been considered a problem in collecting data using surveys. Although participant carelessness cannot be disputed to exist, the impact it has on data quality or the level of influence or bias it produces in results is questionable. The main purpose of this paper is to determine whether participant carelessness is a substantial problem that significantly influences or biases the results of statistical analyses. This is accomplished by analyzing established management relationships through a comparison of the full, careful, and careless samples to determine the impact participant carelessness has on data results regarding correlations, t‑tests, and simple linear regressions. Four detection approaches were used to identify careless participants individually, in pairs, and in three method combinations. The second purpose of this paper is to use the resampled individual reliability (RIR) approach to detect careless participants and compare it to the individual reliability approach to determine whether the two approaches are fundamentally similar. Data were collected using Mechanical Turk (N = 678). Based on the findings, participant carelessness does not appear to be a substantial problem or demonstrate levels of bias in the results in this study. There are two significant differences between the full and careful samples with the t‑tests and the regression comparisons of fit statistics demonstrate the careful samples to have a weak improvement over the full sample; however, none indicate bias. The findings also suggest that the individual reliability and the RIR approaches are not entirely fundamentally similar.