Abstract
Traditional 3D human pose estimation methods often rely on privacy-invasive RGB cameras or signal-sensitive technologies like Wi-Fi and millimeter-wave sensing. Acoustic sensing offers a compelling alternative, yet existing methods are frequently limited by a strict "line-of-sight" requirement, where accuracy drops significantly if the subject is not directly between the audio source and receiver. This paper presents a novel framework that leverages adversarial learning and temporal modeling to achieve position-robust 3D pose estimation. By extracting position-invariant features and accounting for acoustic reverberation, the proposed system maintains high accuracy even when subjects are positioned off-axis, outperforming existing acoustic and radio-frequency baselines.
Introduction
3D human pose estimation is a critical component for applications ranging from healthcare monitoring to smart home interaction. While RGB cameras provide high fidelity, they raise significant privacy concerns and fail in low-light conditions. Alternatives such as Wi-Fi or millimeter-wave (mmWave) sensing are promising but face challenges: they can be obstructed by common materials like water or metal and are often restricted in sensitive environments.
Acoustic sensing represents a "middle ground"—it is non-invasive, functional in total darkness, and utilizes safe, low-frequency waves. However, previous state-of-the-art acoustic methods relied heavily on the "shadow" effect, where the human body physically obstructs the direct sound path. When a person moves even a few dozen centimeters away from this direct line, these models fail to interpret the subtler signals caused by sound reflections and diffraction. This research addresses this limitation, moving beyond simple obstruction to capture the complex geometry of sound bouncing off the human body.
Methodology
The proposed framework treats acoustic pose estimation as a challenge of extracting features that are invariant to the subject's specific coordinates. The approach is built upon three technical pillars:
1. Adversarial Position Discriminator
To ensure the model is robust to where a person stands, the researchers employed a "min-max" adversarial strategy. A discriminator module attempts to predict the subject's specific location based on internal features. Simultaneously, the pose estimation module is trained to generate features that are so "location-blind" that the discriminator cannot identify the subject's coordinates. This forces the network to ignore spatial noise and focus exclusively on the geometry of the human pose.
2. Reverberation-Resistant Temporal Modeling
Because sound travels relatively slowly, the signals reaching a microphone at any given moment contain "echoes" or reflections of the body's position from several milliseconds prior. To account for these delays and the resulting reverberation, the authors introduced a temporal reference window. This window analyzes 16 frames of acoustic data (approximately 1.2 seconds) prior to the target estimation time, providing the necessary context to "de-reverb" the signal and improve accuracy.
3. Phase-Shift Data Augmentation
Collecting real-world acoustic data for every possible floor position is labor-intensive. To overcome data scarcity, the team utilized a phase-shifting technique on the emitted Time-Stretched-Pulse (TSP) signals. By shifting the starting phase of these signals, they effectively tripled the training data without requiring additional physical recording sessions.
Experimental Setup and Datasets
As no existing datasets addressed off-axis acoustic pose estimation, the authors developed a custom dataset in a controlled classroom environment characterized by standard background noise and reverberation.
- Hardware: The setup utilized two loudspeakers and a 4-channel Ambisonics microphone recording in B-Format.
- Data Collection: 3.5 hours of data were recorded from five male subjects performing actions such as walking, squatting, and T-poses.
- Spatial Grid: Subjects were positioned at five lateral intervals: 0cm (on-line), 25cm, 50cm, 75cm, and 100cm away from the center line between the speakers and microphone.
- Ground Truth: High-precision pose data was captured using a 16-camera OptiTrack motion capture system.
Results and Discussion
The proposed method demonstrated significant improvements over existing acoustic and Wi-Fi-based baselines.
- Accuracy: The system achieved a Root Mean Squared Error (RMSE) of 0.53, a substantial improvement over the previous best acoustic method (Shibata et al.), which recorded an RMSE of 0.66.
- Reliability: The Percentage of Correct Keypoints ([email protected]) reached 0.60, compared to the 0.53–0.55 range of baseline models.
- Qualitative Performance: The model proved particularly adept at identifying complex arm positions, such as the "T-pose," even when the subject was 100cm off-axis. This suggests the model successfully learned to interpret subtle acoustic bounce-back rather than relying on simple sound obstruction.
Limitations and Future Work
While the research marks a significant step forward, several challenges remain. The model currently exhibits a "memorization" effect, where it struggles to generalize to floor positions not included in the training grid. Additionally, the system is sensitive to environmental changes; moving from a classroom to a room with different acoustic properties (e.g., glass walls or heavy carpeting) would likely necessitate retraining. Finally, the "wardrobe effect"—where loose clothing alters acoustic reflection properties compared to tight-fitting motion capture suits—can degrade performance.
Future work will focus on developing "room-agnostic" architectures and improving the system's ability to track subjects moving dynamically across a space rather than standing at fixed points.
Summary updated