TL;DR
This paper introduces a novel computer vision approach to automatically analyze human walking patterns in public spaces, specifically to measure how people synchronize their gait to music. By tracking head bobbing motion, they can accurately estimate foot strikes and walking phase, paving the way for large-scale studies on music's influence on behavior.
What Problem Does This Paper Address?
Understanding how environmental factors, like music, influence human behavior in natural settings is a growing area of research. Specifically, the phenomenon of "entrainment," where people unconsciously synchronize their movements to an external rhythm, has been linked to prosocial effects. However, studying this in real-world scenarios is challenging. Traditional methods often require lab settings or intrusive sensors. This paper aims to bridge this gap by developing a non-intrusive video analysis method to quantify walking entrainment to music in public spaces, moving beyond controlled experiments.
Methods
The core of the proposed method is an automated gait characterization algorithm that focuses on the "bobbing" motion of the head during walking. This approach involves several key steps:
- Human Detection: They utilize a deformable parts model (DPM) detector, pre-trained on full-body images, to identify people in the surveillance footage.
- Tracking by Detection: Detections across consecutive frames are linked to form continuous tracks of individuals. This is framed as a multi-object tracking problem solved using a Conditional Random Field (CRF) formulation.
- Motion Estimation: Instead of directly analyzing position, the method estimates motion using a robust multi-resolution motion estimation technique. This is applied to detected body parts to derive an affine motion model.
- Bobbing Estimation: The key innovation lies in characterizing the oscillatory head motion (bobbing) as a sinusoidal function. They fit two independent sinusoidal functions to the x and y components of the head's motion vectors. This fitting process involves:
- Subtracting a first-order polynomial to compensate for perspective effects.
- Employing a non-linear least squares optimization, robustly initialized and iterated to avoid local minima.
- Foot Strike Estimation: The paper hypothesizes that foot strikes occur at the local maxima and minima of the estimated head bobbing motion. These instants are identified by finding the points where the derivative of the fitted sinusoid is zero.
Datasets
The study was conducted in a tunnel in Brighton, UK, equipped with three CCTV cameras and a sound system.
- Collected Dataset: 47 short video clips (under 2 minutes each) of pedestrian traffic were collected. These clips were specifically chosen to study entrainment under two music conditions with different tempos, using the same music genre.
- Annotation: The dataset was manually annotated in two ways:
- Frames where a person's body crossed specific white lines on the floor were marked to measure average speed.
- Heel strike frames were manually identified, excluding obscured footfalls.
- Automated Data: The system processed these clips to generate automated footfall data.
Key Insights and Ideas
- Bobbing as a Proxy for Gait: The paper's central idea is that the subtle, oscillatory head bobbing motion during walking is a strong indicator of gait phase and frequency, and can be reliably extracted even from surveillance footage.
- Robust Sinusoidal Fitting: The method's robustness is enhanced by fitting a sinusoid to the motion data, which is less susceptible to noise and localization inaccuracies compared to direct position analysis. The iterative optimization with geometric progression further improves reliability.
- Foot Strikes from Derivatives: Identifying foot strikes by finding the zero crossings of the derivative of the head bobbing sinusoid is an elegant way to leverage the cyclical nature of walking.
- Public Space Analysis: Demonstrating the feasibility of this analysis in a real-world, public setting (a tunnel) without requiring participants to wear sensors or follow specific paths is a significant step towards ecologically valid behavioral studies.
Key Results
- Foot Strike Estimation Accuracy: The automated foot strike estimation achieved an accuracy of 86.7% when allowing for a 3-frame tolerance (k=3) between manual annotation and automatic detection. With a stricter tolerance (k=1), the accuracy was 56.4%, suggesting that the manual annotations might have some imprecision.
- Tracking Performance: The tracking algorithm showed high performance, with only 11% of tracks being incoherent. The remaining 97% of tracks were stable, with no ID switches or fragmentation, although some tracks had missing parts due to low resolution or distortion.
- Walking Speed Modulation: Preliminary results using manual annotations showed that faster tempo music led to significantly quicker travel through the tunnel compared to slower tempo music (mean velocity 1.27 m/sec vs. 1.19 m/sec, P < 0.05). This aligns with previous findings on music's influence on shopper behavior.
Limitations and Future Work
- Annotation Imprecision: The paper acknowledges that the manual annotations for foot strikes might have some imprecision, which affects the accuracy metrics.
- Track Fragmentation: While generally good, tracks could be fragmented in areas with low resolution or high distortion, though the length was often sufficient for bobbing estimation.
- Future Work: The authors plan to continue refining the model for automatic speed estimation and improve accuracy. They also aim to conduct further data collections for more in-depth music entrainment analysis, exploring the broader features of music beyond just tempo. The potential for music to support safer environments is also a noted area for future investigation.
Summary updated