Abstract
Interest in physical therapy and individual exercises such as yoga/dance has increased alongside the well-being trend, and people globally enjoy such exercises at home/office via video streaming platforms. However, such exercises are hard to follow without expert guidance. Even if experts can help, it is almost impossible to give personalized feedback to every trainee remotely. Thus, automated pose correction systems are required more than ever, and we introduce a new captioning dataset named FixMyPose to address this need. We collect natural language descriptions of correcting a “current” pose to look like a “target” pose. To support a multilingual setup, we collect descriptions in both English and Hindi. The collected descriptions have interesting linguistic properties such as egocentric relations to the environment objects, analogous references, etc., requiring an understanding of spatial relations and commonsense knowledge about postures. Further, to avoid ML biases, we maintain a balance across characters with diverse demographics, who perform a variety of movements in several interior environments (e.g., homes, offices). From our FixMyPose dataset, we introduce two tasks: the pose-correctional-captioning task and its reverse, the target-pose-retrieval task. During the correctional-captioning task, models must generate the descriptions of how to move from the current to the target pose image, whereas in the retrieval task, models should select the correct target pose given the initial pose and the correctional description. We present strong cross-attention baseline models (uni/multimodal, RL, multilingual) and also show that our baselines are competitive with other models when evaluated on other image-difference datasets. We also propose new task-specific metrics (object-match, body-part-match, direction-match) and conduct human evaluation for more reliable evaluation, and we demonstrate a large human-model performance gap suggesting room for promising future work. Finally, to verify the sim-to-real transfer of our FixMyPose dataset, we collect a set of real images and show promising performance on these images.
Introduction
The rise of home-based fitness and remote physical therapy has created a significant need for automated expert guidance. While current AI can estimate poses by identifying joint locations, it often fails to provide actionable, natural language feedback. Effective pose correction requires more than just coordinate mapping; it necessitates spatial reasoning relative to the environment (e.g., "move your hand closer to the lamp"), an understanding of egocentric perspectives (the user's left vs. the camera's left), and the ability to parse linguistic nuances like analogies (e.g., "as if you are gesturing for someone to stop").
The FixMyPose Dataset
To bridge this gap, researchers at UNC Chapel Hill introduced FIXMYPOSE, a large-scale multilingual dataset. It contains 7,691 image pairs and 11,127 descriptions in both English and Hindi. The dataset is designed for diversity, featuring 25 distinct 3D environments and 6 character avatars balanced across demographics to mitigate machine learning biases.
The dataset's descriptions are rich in linguistic complexity:
- Egocentricity: 100% of descriptions use egocentric relations, requiring models to learn the character's orientation.
- Analogous References: Approximately 18% of descriptions use analogies, demanding commonsense knowledge of human gestures.
Methodology and Tasks
The authors define two primary benchmarks:
- Pose Correctional Captioning: Generating a natural language description of the movement required to transition from a "current" pose to a "target" pose.
- Target Pose Retrieval: Selecting the correct "target" pose from a set of distractors, given a "current" image and a correctional description.
The proposed baseline is a multimodal encoder-decoder architecture. It utilizes ResNet for visual features and linear layers for 3D skeletal joint data. A "CA-Stack" (Cross-Attention Stack) aligns these features with language embeddings. For the captioning task, Reinforcement Learning (SCST) with the CIDEr metric as a reward was used to enhance text quality.
Key Insights and Results
- Skeletal Data is Crucial: Incorporating 3D joint data significantly improved performance over image-only models, helping the AI focus on specific body deltas.
- Multimodal Advantage: Vision+Language (V+L) models consistently outperformed unimodal baselines across metrics like BLEU, CIDEr, and METEOR.
- The Human-Model Gap: A stark performance difference exists in the retrieval task: the best model achieved ~38% accuracy, while humans reached 96%. This underscores the difficulty of fine-grained spatial reasoning for current AI.
- Sim-to-Real Transfer: Models trained on synthetic data showed promising generalization to real-world human photos, with only a moderate drop in automated metrics (e.g., CIDEr from 9.91 to 7.12).
Conclusion and Future Directions
While FIXMYPOSE represents a major step forward, several challenges remain. Future work should expand demographic diversity (body types, ages, heights) and transition from static image pairs to video-to-video correction to capture movement dynamics. Additionally, safety is a paramount concern; imperfect AI instructions could lead to injury, necessitating the development of confidence scores or fail-safe mechanisms for real-world deployment.
Summary updated