FixMyPose: Pose Correctional Captioning and Retrieval

Hyounghun Kim, Abhay Zala, Graham Burri, Mohit Bansal

Abstract

Interest in physical therapy and individual exercises such as yoga/dance has increased alongside the well-being trend. However, such exercises are hard to follow without expert guidance (which is impossible to scale for personalized feedback to every trainee remotely). Thus, automated pose correction systems are required more than ever, and we introduce a new captioning dataset named FixMyPose to address this need. We collect descriptions of correcting a "current" pose to look like a "target" pose (in both English and Hindi). The collected descriptions have interesting linguistic properties such as egocentric relations to environment objects, analogous references, etc., requiring an understanding of spatial relations and commonsense knowledge about postures. Further, to avoid ML biases, we maintain a balance across characters with diverse demographics, who perform a variety of movements in several interior environments (e.g., homes, offices). From our dataset, we introduce the pose-correctional-captioning task and its reverse target-pose-retrieval task. During the correctional-captioning task, models must generate descriptions of how to move from the current to target pose image, whereas in the retrieval task, models should select the correct target pose given the initial pose and correctional description. We present strong cross-attention baseline models (uni/multimodal, RL, multilingual) and also show that our baselines are competitive with other models when evaluated on other image-difference datasets. We also propose new task-specific metrics (object-match, body-part-match, direction-match) and conduct human evaluation for more reliable evaluation, and we demonstrate a large human-model performance gap suggesting room for promising future work. To verify the sim-to-real transfer of our FixMyPose dataset, we collect a set of real images and show promising performance on these images.

Introduction: The Challenge of Automated Pose Correction

With the rise of home-based exercise and remote physical therapy, many people follow video streams without expert supervision. While current AI can perform "pose estimation" (identifying where your limbs are), it struggles to provide actionable, conversational feedback to correct those poses. Existing datasets for image-difference captioning usually focus on simple object changes, such as a bird moving in a scene. Correcting a human pose is significantly more complex: it requires an understanding of spatial relationships between body parts, egocentric perspectives (knowing the character's "left" vs. the camera's "left"), and the ability to use environmental landmarks or analogies.

The FixMyPose task requires models to identify differences between a current pose and a target pose to generate instructional feedback.

The FixMyPose Dataset

The core contribution of this work is the FIXMYPOSE dataset, which consists of 7,691 image pairs and 11,127 correctional descriptions. To ensure robustness and avoid machine learning biases, the dataset maintains a balance across characters with diverse demographics performing a variety of movements.

The dataset features 25 diverse 3D environments, including varied rooms like this brick-walled space, to test spatial reasoning.

Key features of the dataset include:

  • Diversity: 25 realistic 3D environments (offices, bathrooms, living rooms) and 6 character avatars.
  • Multilingual Support: Descriptions are provided in both English and Hindi.
  • Linguistic Complexity: 100% of the descriptions use egocentric relations. Humans often use analogies (e.g., "move as if you are gesturing for someone to stop") to describe complex movements, requiring the AI to possess "physical commonsense."

Methodology: Multimodal and Multilingual Learning

The researchers proposed a multimodal encoder-decoder framework to tackle two primary tasks: Pose-Correctional-Captioning (generating instructions) and Target-Pose-Retrieval (selecting the correct target image from 10 candidates).

Technical components include:

  • Multimodal Fusion: A "CA-Stack" (Cross-Attention Stack) integrates visual features from ResNet-101, 3D body joint data (the skeleton), and language embeddings.
  • Reinforcement Learning (RL): For captioning, the model employs Self-Critical Sequence Training (SCST) to optimize for the CIDEr metric directly.
  • Multilingual Sharing: A parameter-sharing setup allows the model to support both English and Hindi efficiently by sharing most layers while keeping word embeddings language-specific.

Results and Key Insights

The study revealed several critical findings regarding the current state of AI in physical instruction:

  1. Multimodal Superiority: Models using both Vision and Language (V+L) significantly outperformed unimodal baselines. In the retrieval task, V+L achieved ~38% accuracy, compared to a random-selection baseline of ~10%.
  2. The Human-Model Gap: There remains a massive performance gap. Humans achieve 96% accuracy on the retrieval task, while the best model sits below 40%. This highlights FIXMYPOSE as a challenging new benchmark.
  3. Importance of 3D Joints: Adding 3D joint data consistently improved captioning scores, proving that raw pixels alone are insufficient to capture the nuances of human movement.
  4. Sim-to-Real Transfer: Models trained on synthetic 3D characters generalized well to real-world photos, suggesting the dataset is a viable proxy for real-world applications.

To better evaluate these models, the authors introduced task-specific metrics—object-match, body-part-match, and direction-match—which more accurately reflect whether an instruction identifies the correct limb and direction than standard NLP metrics like BLEU.

Conclusion and Future Work

FixMyPose bridges the gap between computer vision and personalized physical therapy. While the sim-to-real transfer is promising, future work will involve expanding the character pool to include more body types (height, age, etc.) and exploring more complex reward functions for Reinforcement Learning, such as using the physical distance between predicted and target poses as a direct signal. Expanding the dataset to include more languages and multi-step exercise sequences remains a priority for making AI-driven physical guidance truly accessible.

Summary updated

Abstract

Interest in physical therapy and individual exercises such as yoga/dance has increased alongside the well-being trend. However, such exercises are hard to follow without expert guidance (which is impossible to scale for personalized feedback to every trainee remotely). Thus, automated pose correction systems are required more than ever, and we introduce a new captioning dataset named FixMyPose to address this need. We collect descriptions of correcting a "current" pose to look like a "target" pose (in both English and Hindi). The collected descriptions have interesting linguistic properties such as egocentric relations to environment objects, analogous references, etc., requiring an understanding of spatial relations and commonsense knowledge about postures. Further, to avoid ML biases, we maintain a balance across characters with diverse demographics, who perform a variety of movements in several interior environments (e.g., homes, offices). From our dataset, we introduce the pose-correctional-captioning task and its reverse target-pose-retrieval task. During the correctional-captioning task, models must generate descriptions of how to move from the current to target pose image, whereas in the retrieval task, models should select the correct target pose given the initial pose and correctional description. We present strong cross-attention baseline models (uni/multimodal, RL, multilingual) and also show that our baselines are competitive with other models when evaluated on other image-difference datasets. We also propose new task-specific metrics (object-match, body-part-match, direction-match) and conduct human evaluation for more reliable evaluation, and we demonstrate a large human-model performance gap suggesting room for promising future work. To verify the sim-to-real transfer of our FixMyPose dataset, we collect a set of real images and show promising performance on these images.
BibTeX
@inproceedings{kim2021fixmypose,
  author = {Hyounghun Kim and Abhay Zala and Graham Burri and Mohit Bansal},
  title = {{FixMyPose: Pose Correctional Captioning and Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=OLQHHlKwNEX},
}
Figure