AstroPIE-53: Towards Understanding Failure Modes of Human Pose Estimation in Space

Stephanie Käs, Marten Flaig, Lukas Woyke, Osmanalp Ömeroğlu, Felix Krueckel, Onur Alp Çulha, Bastian LeibearXiv 2026 · DOI · arXiv

TL;DR

AstroPIE-53 is a new pilot dataset and benchmark designed to reveal why state-of-the-art human pose estimation (HPE) models struggle in space. By analyzing 53 annotated astronaut videos, the researchers found that microgravity orientations, bulky spacesuits, and extreme lighting cause massive performance drops for models trained on terrestrial data.

What Problem Does This Paper Address?

We’ve gotten very good at tracking human movement on Earth. Whether it's for sports analytics or AR filters, modern computer vision models like YOLO or ViTPose are remarkably robust. However, space is a different beast.

Astronauts operate in microgravity, meaning there is no "up" or "down." They wear bulky Extravehicular Mobility Units (EMUs) that obscure the human silhouette, and they work in environments with harsh, high-contrast lighting or deep shadows. Most AI models are trained on datasets like COCO, where people are generally upright and wearing normal clothes. This paper addresses the "domain shift" problem: how do these Earth-trained models perform when deployed in the final frontier? Until now, a lack of public, high-quality annotated data has made a systematic failure analysis nearly impossible.

Methods

The authors developed a semi-automated three-stage pipeline (Search, Synchronization, and Annotation) to curate AstroPIE-53. Because of copyright restrictions on raw mission footage, they release "reconstruction scripts" and video identifiers rather than the videos themselves—allowing researchers to rebuild the dataset locally.

To test current technology, they performed a zero-shot benchmark. They took leading models off the shelf—without any space-specific retraining—and tested them against the dataset.

  • For Detection: They tested YOLO26 (fast, convolutional), RF-DETR (transformer-based), and Grounding DINO (a foundation model using text prompts).
  • For Pose Estimation: They evaluated MeTRAbs (3D skeleton regression), NLF (parametric mesh reconstruction), and ViTPose (transformer-based keypoint prediction).

Crucially, they didn't just look at accuracy; they had humans annotate why the models failed, using a taxonomy of "failure tags" like "misaligned legs," "jitter," or "wrong global orientation."

Datasets

The primary contribution is AstroPIE-53, which consists of:

  • 53 videos collected from public space agency footage.
  • Over 20,500 frames with manual bounding box annotations.
  • 14 challenge subsets across six classes: Spacesuit Presence, Environment (Indoor/Outdoor), Orientation, Contrast, Distance, and Occlusion.

The paper notes that while other datasets exist (like SpacesuitPose or the synthetic µPose), they are often too small, not publicly available, or lack the "real-world" messiness of actual mission footage.

Key Insights and Ideas

  • The "Backpack" Problem: Detection models performed reasonably well on frontal views (up to 95% AP) but plummeted to as low as 2% when viewing an astronaut from the back. The large Life Support System (backpack) on the spacesuit completely breaks the "human" silhouette the models expect.
  • Orientation Matters: In microgravity, astronauts are often upside down or horizontal relative to the camera. Earth-trained models, which expect a gravity-aligned vertical axis, frequently misidentify limbs or fail to localize the person entirely.
  • Contrast is a Killer: Space lighting is binary—blindingly bright or pitch black. Low-contrast scenes (common in shadowed areas of the ISS or during EVAs) led to a significant degradation in detection, with Average Precision (AP) dropping below 25% for all models.
  • Foundation Models Lead the Way: Grounding DINO, which uses vision-language alignment (e.g., searching for the prompt "astronaut"), significantly outperformed traditional detectors like YOLO in these challenging zero-shot scenarios.

Key Results

  • Detection: Grounding DINO was the most robust, achieving ~87% AP on astronauts without suits but dropping to ~46% when suits were present. In "Far" distance scenarios, performance for most models was abysmal (often <1% AP).
  • Pose Estimation: No single model dominated. NLF provided the highest proportion of "correct" estimations (69.8%), while ViTPose struggled the most with the domain shift, with 57.2% of its frames being classified as "fully misaligned."
  • Common Failures: "Leg misalignment" was the most frequent partial-body failure (peaking at 8.5% for MeTRAbs), likely due to the bulky, stiff nature of spacesuit lower bodies.

Limitations and Future Work

The authors acknowledge that AstroPIE-53 is a "pilot" dataset. While 20k frames is a great start, it is small compared to terrestrial benchmarks. Furthermore, the dataset lacks 3D ground truth (actual XYZ coordinates for joints), which is nearly impossible to get from historical mission footage without marker-based motion capture.

Future work will likely involve using this failure analysis to design better data augmentation strategies—essentially "teaching" Earth models about microgravity and spacesuits through synthetic data or targeted fine-tuning.

Summary updated

Abstract

Reliable monitoring of astronauts is essential for safety, situational awareness, and autonomous assistance in space missions. While modern computer vision enables accurate human detection and pose estimation on Earth, its robustness in space environments remains largely unquantified due to limited data availability. We present AstroPIE-53, a pilot dataset of 53 annotated astronaut videos collected from publicly available mission footage. To respect copyright restrictions, we release video identifiers, reconstruction scripts, and dense annotations rather than raw videos. The dataset includes bounding boxes, challenge classes, and human-annotated pose failure tags. We benchmark state-of-the-art person detection and pose estimation models under realistic zero-shot deployment conditions and perform the first systematic failure analysis for astronaut monitoring. Our results reveal substantial performance degradation caused by occlusions, unusual body orientations, low contrast, and microgravity motion patterns.
BibTeX
@misc{kas2026astropie,
  author = {Stephanie Käs and Marten Flaig and Lukas Woyke and Osmanalp Ömeroğlu and Felix Krueckel and Onur Alp Çulha and Bastian Leibe},
  title = {{AstroPIE-53: Towards Understanding Failure Modes of Human Pose Estimation in Space}},
  year = {2026},
  eprint = {2611.20207},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20207},
}
Figure