CARE: Counterfactual-Based Algorithmic Recourse for Explainable Pose Correction

Bhat Dittakavi, Bharathi Callepalli, Aleti Vardhan, Sai Vikas Desai, Vineeth N. BalasubramanianWACV 2024

TL;DR

This paper introduces CARE, a novel framework for explainable human pose correction using counterfactual explanations and algorithmic recourse. CARE addresses the limitations of existing pose correction methods by providing actionable, sparse feedback derived from 2D image data, making it more accessible and applicable to real-world fitness and healthcare scenarios.

What Problem Does This Paper Address?

With the rise of home-based fitness and the increasing use of computer vision for monitoring, accurately assessing and correcting human poses has become crucial. While human pose estimation and classification have seen significant advancements, the task of pose correction – identifying incorrect poses and providing guidance to fix them – has received less attention. Existing approaches often rely on expensive 3D data or specialized sensors, are limited to a few specific poses, or offer rudimentary feedback. This paper aims to bridge this gap by developing an explainable, accessible, and effective system for correcting human poses using readily available 2D image data.

Methods

The CARE framework is built upon three core components:

  1. Pose Estimator: An off-the-shelf pre-trained pose estimator (like Mediapipe) is used to extract keypoints from 2D images.
  2. Pose Classifier: The extracted keypoints are converted into pose angle vectors. A shallow, fully connected neural network is trained on these angle vectors to classify the pose into one of several categories. This classifier is crucial for identifying incorrect poses.
  3. Counterfactual Generator and Algorithmic Recourse Module: This is the heart of CARE. When an incorrect pose is detected, the system generates counterfactual explanations (CFEs). These CFEs represent minimal changes to the input (pose angles) that would lead to a correct pose classification. The paper formulates this using an optimization problem that balances minimizing the difference between the current and counterfactual pose (proximity) with maximizing the diversity of generated counterfactuals. Algorithmic recourse then translates the nearest and most optimal counterfactual into a sparse action vector. This vector highlights the specific joint angle adjustments needed for correction, making the feedback actionable and interpretable.

Datasets

The authors evaluate CARE on three diverse datasets:

  • Yoga-20: A subset of the Yoga-82 dataset, focusing on 20 diverse yoga poses.
  • Pilates-32: A dataset of individuals performing 32 Pilates exercises.
  • American Sign Language (ASL) Dataset: Used to demonstrate the framework's applicability beyond fitness, in the domain of hand gesture recognition.

The paper also mentions augmenting existing datasets with artificially generated incorrect poses to create test sets for evaluating pose correction systems.

Key Insights and Ideas

  • Algorithmic Recourse for Pose Correction: Applying algorithmic recourse, a concept typically used for explaining general model decisions, to the specific problem of pose correction is a novel approach. It allows for generating actionable, step-by-step guidance.
  • 2D Data Accessibility: By focusing on 2D image data, CARE significantly lowers the barrier to entry compared to methods requiring 3D sensors or motion capture, making it more practical for widespread use (e.g., via smartphone apps).
  • Sparsity in Actionable Feedback: The generated "action vector" is designed to be sparse, meaning it only suggests changes to the most critical joint angles. This prevents overwhelming the user with too much information and focuses on the essential corrections.
  • Diversity in Counterfactuals: Generating multiple counterfactuals allows for flexibility in how a pose can be corrected, catering to slight variations in user flexibility or preferred correction pathways.
  • Weighted Pose Correction Error (WPCE): The introduction of a new evaluation metric that penalizes changes to already correct joint angles, while still considering the overall correction quality, provides a more nuanced assessment of pose correction systems.

Key Results

  • High Pose Classification Accuracy: The pose classifier component achieved high Top-1 accuracy across all datasets, with scores above 90% (e.g., 93.7% for Yoga-20, 97.7% for ASL, 94.8% for Pilates).
  • Superior Pose Correction Performance: CARE consistently outperformed baseline methods (Centroid, Medoid, Decision Tree, ANN Regression) in terms of Percentage of Corrected Poses (PCP), Mean Absolute Difference (MAD), and the newly introduced Weighted Pose Correction Error (Weighted PCE), especially at lower error thresholds.
  • Qualitative Examples: The paper provides visual examples (Figure 3 and 5) demonstrating how CARE generates sparse action vectors to correct poses like Half-Moon and Low-Lunge, highlighting the specific joint angle adjustments.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the pre-trained pose estimator might not be perfectly trained on the specific datasets, CARE is designed to handle some noise.
  • The generation of incorrect poses is partially automated, and the paper suggests that developing more sophisticated methods for generating realistic incorrect poses based on human flexibility constraints would be beneficial.
  • Future Work:
    • Developing automated methods for generating a larger variety of incorrect poses, specifically considering human flexibility constraints.
  • Improving the robustness of the pose classifier to generate more accurate and effective counterfactuals.
  • Further exploration of the multi-variant counterfactuals to offer more diverse correction options.

Summary updated

BibTeX
@inproceedings{dittakavi2024care,
  author = {Bhat Dittakavi and Bharathi Callepalli and Aleti Vardhan and Sai Vikas Desai and Vineeth N. Balasubramanian},
  title = {{CARE: Counterfactual-Based Algorithmic Recourse for Explainable Pose Correction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure