Collecting big datasets of human activity one checkin at a time

Theus Hossmann, Christos Efstratiou, Cecilia MascoloHotPlanet@MobiSys 2012 · DOI

TL;DR

This paper introduces "up2," a mobile application designed to address the critical need for large-scale, ground-truth-labeled human activity datasets. By leveraging the familiar "check-in" concept from social networks, up2 incentivizes users to contribute sensor data, overcoming the challenges of privacy, energy consumption, and accurate labeling in real-world scenarios.

What Problem Does This Paper Address?

The proliferation of smartphones with rich sensor capabilities has opened doors for applications that infer user activity and location to provide more intelligent services. However, developing and validating these applications requires extensive datasets of sensor traces (accelerometer, GPS, microphone, etc.) meticulously labeled with corresponding user activities. The primary hurdle has been the difficulty in collecting such data outside controlled lab environments, especially in a way that accurately captures ground truth without being overly burdensome to users. Existing datasets are often small-scale, short-lived, and lack comprehensive ground truth, hindering the development of robust, real-world activity detection systems.

Methods

The core innovation of this paper is the "check-in" based data collection strategy implemented in the "up2" mobile application. Instead of continuous, passive sensing, up2 prompts users to manually "check in" to their current activity (e.g., "riding a bicycle," "having dinner"). This check-in acts as a trigger for the app to collect sensor data for a short duration.

Key methodological aspects include:

  • Leveraging the Check-in Concept: Borrowing from successful location-based services like Foursquare, the check-in provides a user-initiated mechanism for data collection and ground truth annotation. This makes the process more deliberate and less intrusive than constant background sensing.
  • Contextualized Activity Selection: Users first select a broad context (e.g., "Work," "Home/Living") and then a specific activity from a pre-defined, context-dependent list. This two-step process balances the need for detailed ground truth with user usability.
  • Sensor Fusion: The app utilizes the accelerometer, microphone, and location services (GPS, cell tower, access point) to capture a rich set of environmental and motion data.
  • Energy Efficiency: To mitigate battery drain, sensor data is cached locally and offloaded to the backend only when a Wi-Fi connection is available, rather than relying on potentially costly 3G data.
  • Incentive Mechanisms: The paper discusses the importance of social and gamified incentives (e.g., leaderboards, badges, friend interactions) to encourage long-term user engagement and data contribution.

Datasets

The paper does not present a pre-existing dataset for evaluation. Instead, the primary contribution is the up2 application itself, designed to generate a novel, large-scale, ground-truth-labeled dataset of mobile phone sensor data. The authors aim to collect data from a large number of users over an extended period, covering a wide range of activities and environments.

Key Insights and Ideas

  • The "Check-in" as a Data Collection Catalyst: The check-in is a powerful, user-friendly paradigm that can transform passive sensing into an active, ground-truth-rich data collection process. It shifts the burden of annotation from complex post-hoc analysis to immediate user input.
  • Balancing Granularity and Usability is Crucial: For widespread adoption, the process of providing ground truth must be extremely simple. The paper highlights the trade-off between collecting highly detailed activity metadata and maintaining a low-friction user experience.
  • Incentives are Key to Long-Term Engagement: Simply collecting data isn't enough; users need compelling reasons to keep the app running and contributing. The paper emphasizes the need for social and gamified elements to drive sustained participation.
  • Wi-Fi Offloading is Essential for Battery Life: For any mobile sensing application to be practical, energy efficiency is paramount. Caching and Wi-Fi-based offloading are presented as critical strategies for managing data transmission costs and battery drain.

Key Results

The paper is primarily a proposal and a system description, not an empirical study with quantitative results on activity detection accuracy. However, it outlines the design and functionality of the up2 prototype.

  • Prototype Functionality: The up2 Android application allows users to authenticate via Google Accounts, perform check-ins with a two-step context/activity selection, and optionally add comments.
  • Check-in Effort: The check-in process is designed to be quick, requiring approximately 5 clicks and taking about 10 seconds, comparable to existing location-based check-in services.
  • Sensor Data Collection: The system collects accelerometer, microphone, and location data.
  • Data Offloading: Sensor data is cached locally and transmitted via Wi-Fi to conserve mobile data.

Limitations and Future Work

Limitations:

  • Ground Truth Reliability: The authors acknowledge that user-reported data can be unreliable due to mislabeling (intentional or unintentional), system failures, or users not checking in at the exact moment of the activity.
  • Limited Initial Sensor Set: The current prototype uses only accelerometer, microphone, and location. Future work will incorporate additional sensors like Wi-Fi, Bluetooth, and gyroscope.
  • Initial Incentive Strategy: While incentives are discussed, the paper doesn't present a fully implemented or evaluated incentive system.

Future Work:

  • Sanity Checks for Data Reliability: Implementing backend sanity checks based on sensor data (e.g., verifying high accelerometer values for "running") to detect erroneous labels.
  • Iterative Deployment and Learning: Using collected data to "learn" optimal sampling schemes for different activities and to develop smarter activity detection algorithms that can predict activities before a check-in.
  • Expanding Sensor Integration: Incorporating Wi-Fi, Bluetooth, and gyroscope data for more comprehensive sensing.
  • Decoupling User Accounts: Moving away from Google Account dependency for authentication.
  • Controlled Deployment and User Studies: Initially deploying the application in a controlled, invitation-based environment to refine the user interface and incentive mechanisms before a wider public release.
  • Developing Activity Detection Algorithms: Using the collected dataset to train and evaluate advanced activity and context detection algorithms.

Summary updated

BibTeX
@inproceedings{hossmann2012collecting,
  author = {Theus Hossmann and Christos Efstratiou and Cecilia Mascolo},
  title = {{Collecting big datasets of human activity one checkin at a time}},
  booktitle = {HotPlanet@MobiSys},
  year = {2012},
  doi = {10.1145/2307836.2307842},
}
Figure