Abstract
BGM2Pose is a first-of-its-kind system that transforms everyday background music (BGM) into an active sensing signal for 3D human pose estimation. By employing contrastive learning and frequency-specific attention, the model can reconstruct human movement from subtle sound reflections. This approach offers a privacy-preserving, non-invasive alternative to camera-based systems, functioning effectively even in total darkness or through occlusions.
Introduction
Traditional 3D human pose estimation typically relies on RGB cameras. However, these systems face significant challenges, including privacy concerns, failure in low-light environments, and an inability to see through physical occlusions like furniture. To address these issues, researchers have explored "active sensing" using WiFi, radio waves, or acoustics.
Acoustic sensing is particularly attractive due to the ubiquity and safety of sound. However, previous acoustic methods relied on chirp signals—high-pitched, repetitive beeps that are disruptive and annoying to human occupants. The primary challenge in replacing chirps with regular music (BGM) is that music is "non-stationary." Its volume and pitch fluctuate constantly, creating dynamic changes that are orders of magnitude louder than the minute sound reflections caused by a moving human body.
Methodology
The BGM2Pose framework treats BGM as a probe to map the environment. The architecture consists of three primary innovations designed to extract pose information from the complex acoustic environment.
1. Acoustic Feature Engineering
The system records audio using an ambisonics microphone to capture 3D sound directionality. It then subtracts the original "clean" music track from the recorded "noisy" track within the log-mel spectrogram domain. This process calculates a "pseudo transfer function," which highlights how the physical environment and the human subject have altered the sound.
2. Frequency-wise Attention (FA) Module
Unlike chirps that cover a broad, constant frequency range, music occupies specific bands at varying times. The FA module dynamically identifies which frequency bands currently contain the most useful pose-related information, allowing the model to ignore "silent" or irrelevant portions of the spectrum.
3. Contrastive Pose Extraction (CPE) Module
To separate the "song" from the "pose," the authors utilize contrastive learning. The model is trained to align recorded audio features with their corresponding 3D poses in a shared mathematical space. By using hard negative sampling—presenting examples where the music is identical but the pose differs—the network is forced to ignore the melodic content and focus strictly on physical reflections.
Datasets and Evaluation
The researchers introduced the AMPL (Acoustic Music-based Pose Learning) dataset to benchmark the system. This dataset links four BGM tracks (Ambient and Jazz genres) with ground-truth 3D poses captured via a high-end Motion Capture (Mocap) system. It includes multiple subjects performing various actions such as standing, bending, and walking in a real-world classroom environment. Additionally, an "In Plain Clothes" dataset was recorded to test robustness in scenarios where subjects wore regular clothing instead of Mocap suits.
Key Results
BGM2Pose significantly outperformed existing baselines, including those adapted from WiFi-sensing and chirp-based acoustic sensing:
- Accuracy: In a single-music setting, it achieved a Root Mean Square Error (RMSE) of 0.923, compared to 1.090 for the best previous acoustic method.
- Generalization: The system maintained high accuracy (RMSE 1.036) even when tested on music tracks not encountered during training.
- Robustness: The model remained stable under real-world noise conditions (SNR=10) and successfully reconstructed poses for individuals in regular clothing, which can absorb sound differently than specialized suits.
The study also noted that genre impacts performance; Jazz proved more challenging than Ambient music due to frequent "silent" intervals that provide no sensing signal.
Conclusion and Future Work
BGM2Pose demonstrates that background music can be effectively repurposed as a sensing tool, turning "noise" into a functional illumination source. While the system marks a major advancement, limitations remain. Current performance is optimized for line-of-sight positioning between speakers and microphones, and the model occasionally struggles with symmetrical movements like walking. Future research will focus on making the system "room-agnostic," allowing for deployment in diverse environments without the need for specific recalibration.