Identification of Careless Responding in Ecological Momentary Assessment Research: From Posthoc Analyses to Real-Time Data Monitoring

With the emerging ubiquity of cell phones, ecological momentary assessment (EMA) as a set of methods
enable researchers to study momentary social, psychological, and affective responses to everyday
life. Additionally, EMA enables researchers to acquire longitudinal data without the need for multiple
lab visits. As the use of EMA in research increases, so too does the necessity of determining what constitutes
valid or careless individual EMA responses to ensure validity and replicability of findings.
Currently, EMA studies solely consider the response rate of a participant for exclusion. Yet, other features
of an assessment can help to determine whether a response is careless or implausible. Here, we
examined over 18,000 EMA text message responses of individual affect items to derive a data-driven
model of what constitutes a “careless response.” Results from this study indicate that an overly fast time
to complete items (≤ 1 s), an overly narrow within assessment response variance (SD ≤ 5), and the percentage
of items that fall at the mode (≥ 60%) are independent and reliable indicators of a careless
response. Excluding careless responses such as these remove implausible positive correlations among
psychometric antonyms (e.g., relaxed and anxious). Further, by identifying and removing careless
responses, we also identify careless responders, participants who could be removed from group analyses.
We use these results to develop and introduce an R package, EMAeval, so EMA researchers may
similarly identify careless responses and responders either online during data collection or posthoc, after
data collection has completed.