This paper, presented at NeurIPS 2023, introduces a novel approach to human pose estimation that moves away from the camera lens and into the fabric of our clothes. Researchers from the University of British Columbia and Texavie Technologies have developed "Intelligent Knee Sleeves" to track 3D lower body motion using a combination of smart textiles and machine learning.
The Problem with Traditional Pose Estimation
Traditional 3D human pose estimation is heavily dominated by computer vision. While effective, camera-based systems face three major hurdles:
- Occlusion and Field of View: If a limb is hidden behind an object or moves out of the camera's frame, the system fails to track it accurately.
- Privacy Concerns: Many users are uncomfortable with constant video monitoring, especially in private spaces like homes or rehabilitation clinics.
- Environmental Constraints: Cameras require specific lighting conditions and fixed setups to work reliably.
Wearable alternatives, such as standalone Inertial Measurement Units (IMUs), offer a privacy-preserving solution but often suffer from "drift." This occurs when small measurement errors accumulate over time, leading to wildly inaccurate position estimates. This research seeks to bridge the gap by fusing IMU data with textile-integrated pressure sensors that "feel" muscle contractions, providing a more stable and private kinesthetic sensing modality.
Hardware and Setup
The team utilized Texavie MarsWear Knee Sleeves, which are constructed from stretchable, washable double-jersey rib fabrics.
Each sleeve integrates 14 channels of yarn-shaped pressure sensors—strategically positioned around the hamstrings, quads, calves, and shins—alongside 9 channels of IMU data.
To train and validate their system, the authors introduced a comprehensive new multimodal dataset. It contains over 140,000 synchronized frames collected over 12 days. Ground truth data was labeled using a professional-grade Optitrack Motion Capture (MoCap) system with six cameras and reflective markers. The dataset covers common exercises including squats, hamstring curls, and leg raises, and includes various subjects wearing different sizes of knee sleeves to test the model's ability to generalize.
Methods: The Stacked LSTM Model
To process the complex, time-series data generated by the sleeves, the researchers implemented a Stacked Long Short-Term Memory (LSTM) network. LSTMs are particularly well-suited for this task because they excel at processing sequential data where a current movement heavily depends on previous states.
The model takes a sliding window of 250 sensor samples as input. It then outputs quaternions for eight major lower-body joints. Quaternions are used instead of Euler angles because they provide a robust mathematical way to represent 3D rotations without suffering from "gimbal lock."
To handle the inherent "jitter" and uneven time intervals typical of Bluetooth data transmission, the researchers used Fourier resampling. This crucial signal processing step aligns the wearable data perfectly with the MoCap ground truth.
Key Insights
- Proprioception via Pressure: The core innovation of this system is that muscle contraction, detected by the yarn sensors, provides vital context that IMUs alone lack. When a user squats, their muscles fire in a specific pattern. The pressure sensors capture this "intent" and "effort," which helps the model correct the drift errors typical of IMUs.
- Kinematic Extrapolation: Surprisingly, a sleeve worn only on the knee can accurately predict the movement of the hip and ankle. Because human lower-body movements are kinematically linked, the hip must move in a predictable way for the knee to reach a certain angle during an exercise like a squat.
- Privacy-First Design: By operating entirely camera-free, the system enables high-fidelity motion tracking in settings where video is inappropriate, opening doors for remote home healthcare and private physical therapy.
Results and Future Work
The system achieved an impressive average error of just 7.21 degrees across all eight lower body joints when compared to the MoCap ground truth. Furthermore, the model performed well on "unseen" tasks and individuals, proving it learns general human kinematics rather than just memorizing specific users' movements. Knee angle prediction was the most accurate, while toe angle showed the least error simply because it remains relatively stable during the selected exercises.
Despite these strong results, the authors note several areas for future improvement:
- Soft Tissue Artifacts: The way skin and muscle shift under the sleeve during intense exercise can cause sensor displacement and lead to errors.
- The "Starting Position" Problem: The sensors currently measure relative change. If a user starts in a seated position versus a standing position, the model can become confused without a calibration period to establish an absolute "zero" point.
- Dataset Expansion: The current model is trained on a limited set of exercises. Expanding the dataset to include walking, running, and more complex, dynamic movements is a logical next step for the technology.
Summary updated