Automated Ecological Assessment of Physical Activity: Advancing Direct Observation

Jordan Carlson, Bo Liu, James F. Sallis, Jacqueline Kerr, J. Aaron Hipp, Vincent S. Staggs, Amy Papa, Kelsey Dean, Nuno Vasconcelos

TL;DR

This paper introduces a novel computer vision system for automatically assessing physical activity in outdoor public spaces, moving beyond manual observation. The system demonstrates promising sample-level validity, accurately estimating sedentary behavior and moderate-to-vigorous physical activity (MVPA) proportions, which are key metrics for public health.

What Problem Does This Paper Address?

Traditional methods for assessing physical activity in public spaces, like parks and schoolyards, rely on direct observation. While valuable, these methods are labor-intensive, time-consuming, and often capture only snapshots of activity, limiting their ability to provide continuous monitoring or rapid feedback for interventions. Existing automated approaches have faced limitations, such as requiring participants to wear specific colored vests or being restricted to controlled indoor environments. This study aims to bridge this gap by developing and validating a computer vision system capable of automated, continuous, and ecological assessment of physical activity in open outdoor settings.

Methods

The core of this research lies in developing and evaluating computer vision algorithms for physical activity assessment. The system utilizes deep convolutional neural networks for action recognition to process video data from three cameras. These algorithms were trained to predict:

  • The number of participants in different activity states (sedentary, light, moderate, vigorous).
  • The proportion of participants in these states.
  • Group-based metabolic equivalents of tasks (MET)-minutes.

The system was designed to be adaptable to outdoor environments, handling variations in lighting and camera angles, and importantly, did not require participants to wear special clothing. Criterion validity was assessed by comparing the computer vision algorithm's output against data from accelerometers worn by participants. Both second-level (1-second intervals) and minute-level aggregated data were analyzed.

Datasets

The study utilized video data collected from three digital cameras in an open outdoor area on the University of California, San Diego campus. This video data was synchronized with accelerometer data from nine participants. The total dataset comprised 6630 seconds of matched accelerometer and video data per camera. For training and testing the computer vision algorithms, blocks of 200 seconds of video were alternately assigned to a training dataset (N = 3830) and a testing dataset (N = 2800). No new benchmark datasets were introduced; rather, existing accelerometer data served as the ground truth.

Key Insights and Ideas

  • Automated Observation is Feasible: The study demonstrates that computer vision can effectively automate the process of observing and quantifying physical activity in real-world outdoor settings, offering a significant advancement over manual methods.
  • MVPA and Sedentary Behavior are Key: The algorithms showed strong validity for estimating sedentary behavior and moderate-to-vigorous physical activity (MVPA) proportions. This is particularly important as these are commonly used metrics in public health research and interventions.
  • Combining Moderate and Vigorous Activity is Beneficial: While the algorithms had mixed results for individual moderate and vigorous activity levels, combining them into MVPA improved overall validity, highlighting the practical utility of this aggregated metric.
  • Outdoor Adaptability is Crucial: The system's design to handle outdoor variability (lighting, camera angles) is a critical step towards broader applicability in diverse public spaces like parks and schoolyards.
  • Sample-Level Validity is Achievable: The system demonstrated acceptable "sample-level" validity, meaning it can reliably estimate activity across a collection of time intervals, which is suitable for ecological assessments of settings.

Key Results

  • Sedentary and MVPA: The number and proportion of participants in sedentary and MVPA categories showed small biases (within 20% of the criterion mean) and excellent intraclass correlation coefficients (ICCs) ranging from 0.82 to 0.98.
  • Total MET-minutes: Total MET-minutes were slightly underestimated by 9.3-17.1%, with good ICCs (0.68–0.79).
  • Light Activity: The algorithms showed poorer performance for light activity, with lower ICCs.
  • Limits of Agreement: While sample-level validity was good, the limits of agreement for individual time intervals were wider than desired (outside the 30% criterion), indicating variability in bias across shorter time scales. This suggests that aggregation of data over longer periods (e.g., 5 minutes) might be necessary for more precise individual-level estimates.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Setting Diversity: The study was conducted in a single, open outdoor space with no obstructions, which limits generalizability to more complex environments with trees, playground equipment, or varied surfaces.
  • Small Sample Size: The number of participants and observation days were relatively small.
  • Prescribed Activities: Participants engaged in specific, prescribed activities, which may not fully reflect the spontaneity of real-world park use.
  • Individual-Level Validity: While sample-level validity was good, the limits of agreement suggest caution when interpreting short-interval data at an individual level.

Future work should focus on:

  • Testing in Diverse Settings: Evaluating the algorithms in a wider range of environments with varying demographics, group sizes, and physical features.
  • Longer Duration Data Collection: To support aggregation over longer time intervals and improve individual-level validity.
  • Investigating Predictors of Bias: Understanding factors that contribute to bias to inform data interpretation.
  • Exploring Other Capabilities: Investigating whether the system can identify demographic information like gender, race/ethnicity, or age group, which is often captured by traditional observation methods.
  • Comparison with Traditional Methods: Directly comparing the accuracy and efficiency of the computer vision system against established direct observation tools like SOPARC.

Summary updated

Abstract

Technological advances provide opportunities for automating direct observations of physical activity, which allow for continuous monitoring and feedback. This pilot study evaluated the initial validity of computer vision algorithms for ecological assessment of physical activity. The sample comprised 6630 seconds per camera (three cameras in total) of video capturing up to nine participants engaged in sitting, standing, walking, and jogging in an open outdoor space while wearing accelerometers. Computer vision algorithms were developed to assess the number and proportion of people in sedentary, light, moderate, and vigorous activity, and group-based metabolic equivalents of tasks (MET)-minutes. Means and standard deviations (SD) of bias/difference values, and intraclass correlation coefficients (ICC) assessed the criterion validity compared to accelerometry separately for each camera. The number and proportion of participants sedentary and in moderate-to-vigorous physical activity (MVPA) had small biases (within 20% of the criterion mean) and the ICCs were excellent (0.82-0.98). Total MET-minutes were slightly underestimated by 9.3-17.1% and the ICCs were good (0.68-0.79). The standard deviations of the bias estimates were moderate-to-large relative to the means. The computer vision algorithms appeared to have acceptable sample-level validity (i.e., across a sample of time intervals) and are promising for automated ecological assessment of activity in open outdoor settings, but further development and testing is needed before such tools can be used in a diverse range of settings.
BibTeX
@article{carlson2017automated,
  author = {Jordan Carlson and Bo Liu and James F. Sallis and Jacqueline Kerr and J. Aaron Hipp and Vincent S. Staggs and Amy Papa and Kelsey Dean and Nuno Vasconcelos},
  title = {{Automated Ecological Assessment of Physical Activity: Advancing Direct Observation}},
  journal = {International Journal of Environmental Research and Public Health},
  year = {2017},
  doi = {10.3390/ijerph14121487},
}
Figure