TL;DR
MOSAIC (Motion-based Sensor Assignment and Inertial Calibration) enables "wear and go" inertial motion capture by automatically identifying sensor placement and orientation from natural movement. By accumulating evidence over time, it eliminates the need for tedious calibration poses and allows high-fidelity 3D reconstruction using existing, frozen pose estimation models.
What Problem Does This Paper Address?
Sparse inertial motion capture (mocap) uses a handful of sensors (usually six) to reconstruct full-body movement. However, these systems typically have a high "friction" cost: the user must specify which sensor is on which limb (assignment) and perform a specific calibration pose (like a T-pose) to align the sensor's coordinate frame with the body's (calibration).
If a sensor slips or is put on slightly crooked, the reconstruction fails. While some methods attempt "online" calibration, they often require known sensor identities or specific movement sequences. The authors of this paper wanted to create a truly calibration-free system where a user can simply strap on six interchangeable sensors in any order and orientation and start moving.
Methods
The core of the paper is the MOSAIC framework, which decouples the persistent sensor setup from the ever-changing body pose.
The system works in three main stages:
- Window-level Evidence Extraction: A neural network processes short windows of unordered IMU data. It uses a "sensor-order-equivariant" architecture (meaning it doesn't care about the initial order of the data streams) with temporal and cross-sensor attention to predict two things: which body segment the sensor is likely on and what its mounting rotation is.
- Persistent Evidence Accumulation: This is the "aha" moment of the paper. Instead of relying on a single-shot guess, MOSAIC maintains a running sum of assignment scores and rotation matrices over the entire session. Because the sensor setup is fixed while the body pose changes, "ordinary" movements that might be ambiguous individually provide complementary evidence over time.
- Canonicalization: Once the setup is estimated (using Hungarian matching for assignment and SVD-based projection for rotation), MOSAIC transforms the raw IMU streams into a "canonical" format. This allows the data to be fed into frozen, pretrained pose estimators (like LSTM, PIP, or DynaIP) without needing to retrain them for uncalibrated data.
Datasets
The authors used a mix of synthetic and real-world data:
- AMASS: A large-scale database of human motion used to generate synthetic IMU measurements for pretraining.
- DIP-IMU: A standard dataset for sparse inertial mocap. Subjects 1–8 were used for fine-tuning, while 9–10 were held out for evaluation.
- TotalCapture: Used as an additional evaluation benchmark to test the system's generalizability.
- Live Demo Data: The authors recorded their own dataset using Xsens IMUs with randomized setups and a RealSense camera for visual ground truth.
Key Insights and Ideas
- Persistence is Key: The fundamental insight is that the sensor-to-segment relationship is a persistent physical parameter. While a single arm swing might not reveal the leg sensor's orientation, a few seconds of walking and moving naturally eventually constrains all degrees of freedom.
- Modular Interface: By treating setup estimation as a "pre-processing" step that canonicalizes data, MOSAIC acts as a universal adapter. This means researchers can improve pose estimation models and setup estimation models independently.
- Evidence Pooling: The use of Matrix Fisher distributions motivates their choice to sum evidence matrices. This mathematical framework allows them to treat overlapping windows of motion as additive evidence rather than independent observations.
Key Results
- Rapid Recovery: In evaluations, MOSAIC recovered correct sensor assignments within 2 to 8 seconds of ordinary motion.
- Near-Calibrated Accuracy: For sensors with bounded mounting offsets (up to 30°), the system achieved pose accuracy nearly identical to systems that were manually calibrated (within ~1° of angular error).
- Robustness to Arbitrary Orientation: Even when sensors were attached with completely random rotations (the "Arbitrary" regime), MOSAIC's accumulation strategy significantly outperformed baselines, reducing error by over 20% compared to single-window estimates.
- Plug-and-Play: The authors demonstrated that MOSAIC successfully "canonicalized" inputs for multiple state-of-the-art pose networks (PIP, DynaIP) without any modifications to those networks.
Limitations and Future Work
While MOSAIC is a major step toward consumer-ready mocap, some limitations remain:
- Prescribed Sites: The system currently assumes sensors are placed at six specific "standard" sites (forearms, shanks, head, pelvis). It cannot yet handle arbitrary placement (e.g., a sensor on the chest instead of the head).
- Rigid Attachment: The model assumes the sensor doesn't move relative to the bone once attached. It does not explicitly model "soft tissue artifact" or sensors sliding down a limb during a long session.
- Future Directions: The authors suggest that future work could explore more flexible sensor layouts and methods to handle time-varying alignment caused by sensor slippage.
Summary updated