Multimodal Human and Environmental Sensing for Longitudinal Behavioral Studies in Naturalistic Settings: Framework for Sensor Selection, Deployment, and Management

Brandon M. Booth, Karel Mundnich, Tiantian Feng, Amrutha Nadarajan, Tiago H. Falk, Jennifer L. Villatte, Emilio Ferrara, Shrikanth NarayananJournal of Medical Internet Research 2019 · DOI

Abstract

The satisfactory compliance rates and minimal impact of unexpected events during the case study suggest that the challenges, criteria, methods, and mitigation strategies presented as a guide for researchers are helpful for sensor selection and management in longitudinal human behavior studies in the wild.

Introduction: The Challenge of "In the Wild" Research

While mobile sensing technologies have enabled behavioral research to move from controlled laboratory environments into naturalistic settings like homes and workplaces, this transition introduces significant variables. Researchers frequently encounter high participant attrition, sensor failures, data loss, and complex ethical dilemmas regarding privacy.

Until now, a unified guide to navigating the trade-offs between data fidelity and participant burden has been missing. This paper addresses this gap by providing a structured roadmap for sensor selection, deployment, and management, ensuring that the resulting data is both high-quality and ethically sound.

A General Data Collection Framework

The authors propose a comprehensive framework that organizes information flow from three sensor categories—environmental, nonwearable trackers, and wearables—through intermediate "data hubs" (such as smartphones) to secure research servers.

The proposed framework highlights the paper's scope within the broader research lifecycle, focusing on sensor selection, study design, and participant concerns.

The core of this approach involves a multi-dimensional evaluation of sensors based on:

  • Research Objectives & Logistics: Aligning signals of interest with specific measurement mechanisms and sampling rates.
  • Sensor Characteristics: Evaluating battery life, physical robustness, and the impact of proprietary "black box" preprocessing by manufacturers.
  • Participant Engagement: Minimizing cognitive load and physical interference to ensure long-term compliance.
  • Human Subject Protection: Implementing "privacy by design," such as on-device anonymization of raw audio features before transmission.

The MOSAIC Case Study

The framework was validated through the MOSAIC case study, a 10-week longitudinal project involving over 200 hospital staff. This study generated a massive multimodal dataset including over 355,000 hours of usable data. Signals collected included cardiac metrics (ECG and PPG), physical activity, sleep, speech features, respiration, and environmental metrics (temperature, humidity, light).

Key Results and Insights

The framework proved highly effective, yielding several critical findings:

  • High Compliance and Low Attrition: Average daily compliance reached 65%, with Fitbit usage specifically peaking at 73.1%. The dropout rate remained remarkably low, under 4%.
  • The "Black Box" Challenge: The authors emphasize the importance of accessing raw data over preprocessed consumer metrics (like "sleep quality") to ensure scientific validity.
  • Logistics as Science: Success often hinges on "boring" details, such as the availability of power outlets or managing international shipping delays for hardware.
  • Privacy via Edge Computing: By extracting emotional features from audio on-device and discarding raw recordings, the team secured IRB approval for sensitive environments like hospitals.

Limitations and Future Directions

Despite its success, challenges remain. High-fidelity sensing is still constrained by mobile battery life, and physical discomfort from wearables (e.g., skin rashes) persists. Future work will focus on automated signal enhancement and remote firmware updates to further reduce the logistical burden on research staff and participants alike.

Summary updated

Abstract

The satisfactory compliance rates and minimal impact of unexpected events during the case study suggest that the challenges, criteria, methods, and mitigation strategies presented as a guide for researchers are helpful for sensor selection and management in longitudinal human behavior studies in the wild.
BibTeX
@article{booth2019multimodal,
  author = {Brandon M. Booth and Karel Mundnich and Tiantian Feng and Amrutha Nadarajan and Tiago H. Falk and Jennifer L. Villatte and Emilio Ferrara and Shrikanth Narayanan},
  title = {{Multimodal Human and Environmental Sensing for Longitudinal Behavioral Studies in Naturalistic Settings: Framework for Sensor Selection, Deployment, and Management}},
  journal = {Journal of Medical Internet Research},
  year = {2019},
  doi = {10.2196/12832},
}
Figure