Introduction
Reconstructing 3D human motion from sparse wearable sensors, such as Inertial Measurement Units (IMUs), is a significant challenge in computer vision and graphics. While dense sensor suits (17+ sensors) provide high accuracy, they are intrusive and costly. Sparse setups using only six sensors—typically placed on the wrists, knees, head, and pelvis—are far more user-friendly but lead to an under-constrained problem where multiple body poses can produce nearly identical sensor signals.
The Ambiguity Challenge
The primary hurdle in sparse IMU reconstruction is semantic ambiguity. For instance, the rotation and acceleration data for a person standing perfectly still may be indistinguishable from those of a person sitting still if only limb data is considered.
As illustrated in Figure 1, identical IMU readings can correspond to distinct actions like "standing" or "sitting." Without additional context, data-driven models often "flicker" between these states or produce physically impossible poses. This paper proposes using textual action descriptions as a semantic "tie-breaker" to resolve these physical ambiguities.
Proposed Framework
The authors introduce a multimodal architecture designed to bridge the gap between raw sensor data and semantic meaning. The system consists of three main components: a Sensor Encoder, a Text Encoder, and a Text-Sensor Fusion Module.
1. Uncertainty-Aware Sensor Encoding
The Sensor Encoder does not treat all data as equally valid. It regresses an uncertainty value for each IMU channel and employs Uncertainty-guided Spatial Attention (UGSA). This allows the model to "trust" sensors with lower uncertainty more when calculating relationships between body parts, making the system robust against hardware noise and drift.
2. Semantic Grounding via Text
Utilizing a frozen CLIP-based architecture, the Text Encoder processes both sequence-level labels (the overall action) and frame-level labels (specific movements). Before final fusion, a contrastive loss ensures that sensor features and text features are aligned in the same mathematical space, ensuring the "signal" for an action matches its semantic "concept."
3. Hierarchical Temporal Transformer (HTT)
The heart of the fusion process is the HTT Module, which captures motion patterns across various time scales. By using windowed self-attention and patch merging, the model can understand both fine-grained movements and the broader temporal structure of an action.
Key Results and Insights
The framework was evaluated on major benchmarks including AMASS, BABEL, DIP-IMU, and Totalcapture. The results demonstrate that text is not merely "extra info" but a critical prior for accuracy:
- Accuracy Improvements: On the Totalcapture dataset, the model reduced the SIP error (global rotation error of limbs) to 7.92°, significantly outperforming the previous state-of-the-art "Transpose" model (12.30°).
- Positional Precision: It achieved a positional error of just 3.70 cm, compared to 4.85 cm for existing methods.
- Qualitative Stability: Visually, the model produces much smoother transitions and avoids the erratic alternation between similar poses that plagues sensor-only models.
Conclusion and Future Work
This research proves that spatial relationships and textual semantics are vital for high-fidelity motion reconstruction from sparse sensors. While the current model achieves state-of-the-art results, future directions include optimizing for real-time performance (currently ~83ms latency) and integrating natural language reasoning to allow the system to understand more complex, unscripted human instructions.
Summary updated