DIOR: Dataset for Indoor-Outdoor Reidentification Long Range 3D/2D Skeleton Gait Collection Pipeline, Semi-Automated Gait Keypoint Labeling and Baseline Evaluation Methods

Yuyang Chen, Praveen Raj Masilamani, Bhavin Jawade, Srirangaraj Setlur, Karthik Dantu

This paper, titled "DIOR: Dataset for Indoor-Outdoor Reidentification," was presented by researchers from the University at Buffalo. It introduces a new framework and dataset designed to push the boundaries of gait recognition—identifying people by how they walk—specifically in challenging long-range and outdoor environments.

TL;DR

The DIOR project introduces a massive dataset of 1.6 million frames and a novel semi-automated pipeline that uses multi-view geometry to label 3D/2D gait skeletons at distances up to 60 meters. This allows for precise tracking even when a person is only 20 pixels tall—a resolution where standard pose estimation algorithms completely fail.

What Problem Does This Paper Address?

Gait recognition is a powerful biometric because it works at a distance where faces are blurry and can be captured from various angles. However, existing datasets have a "Goldilocks" problem: they are either recorded in highly controlled indoor environments with expensive Motion Capture (MoCap) systems or outdoors with low-resolution imagery that lacks accurate ground-truth labels.

The "valley of death" for gait recognition is the long-range outdoor setting. When a subject is 60 meters away, they might occupy a mere 20–25 vertical pixels in an RGB frame. At this resolution, state-of-the-art pose estimators (like AlphaPose or OpenPose) cannot reliably find joints, making it nearly impossible to train or evaluate gait-based Re-Identification (Re-ID) systems for real-world security or UAV applications.

Methods

The authors developed a two-part pipeline to bridge the gap between high-fidelity indoor data and "in-the-wild" outdoor data.

  1. Indoor Pipeline: They used a Vicon MoCap system (which tracks reflective markers with millimeter precision) alongside an array of RGB cameras. By using the Perspective-n-Point (PnP) algorithm, they localized the RGB cameras within the MoCap space, allowing them to project 3D joint labels directly onto 2D video frames with pixel-level accuracy.
  2. Outdoor Pipeline: Since MoCap doesn't work outdoors, they used a "triangulate-and-reproject" strategy. They set up three "close-range" cameras and one "long-range" camera (60m away).
  3. They used standard pose estimators on the high-resolution close-range views.
  4. They triangulated these 2D points into a 3D skeleton.
  5. They then reprojected that 3D skeleton onto the long-range camera's view.
  6. Refinement: To ensure the cameras were perfectly aligned, they used Bundle Adjustment (via the GTSAM library). This mathematical optimization corrected for tiny misalignments in camera angles, reducing reprojection errors significantly.

Datasets

The primary contribution is the DIOR Dataset:

  • Scale: 1.649 million RGB frames.
  • Subjects: 14 individuals with two different sets of clothing each.
  • Labels: 3D and 2D skeleton gait labels.
  • Long Range: Includes 200,000 frames from a camera 60 meters away.
  • Comparison: Unlike previous datasets like CASIA-B or BRIAR, DIOR provides MoCap-grade labels for subjects under 25 pixels in height (see Table 1 in the paper).

Key Insights and Ideas

  • The Failure of Off-the-Shelf Tools: One of the most striking findings was the "Infeasibility of Long-Range Detection." The authors tested AlphaPose, MLKIT, and YOLOv7 on their 60m footage. The success rate for correct joint detection was essentially 0% for the raw long-range frames. This highlights why the DIOR labeling pipeline is so necessary—it provides ground truth where AI currently sees only noise.
  • Geometry as a Labeler: The paper proves that multi-view geometry is a more robust "teacher" than deep learning for labeling low-resolution data. By "transferring" knowledge from high-res views to low-res views through 3D space, they created a dataset that can finally be used to train models to "see" gait in low resolution.
  • Robustness to Environment: The reprojection method is inherently robust to lighting changes and partial occlusions because the "labels" are generated from the geometry of other cameras, not the pixels of the long-range camera itself.

Key Results

  • Label Accuracy: After applying Bundle Adjustment, the authors observed an average of 5-pixel correction in label placement, ensuring the ground truth is highly reliable.
  • Baseline Benchmarking: The authors evaluated three state-of-the-art gait recognition models (GaitGraph, GaitMixer, and GaitFormer) on the DIOR dataset.
  • Rank@1 Accuracy: In cross-domain tests, GaitMixer showed the most robust performance in the long-range "Out-2" scenario, though the relatively high error rates across all models suggest that DIOR is a challenging and necessary benchmark for the field.

Limitations and Future Work

  • Subject Diversity: With only 14 subjects, the dataset is deep (many frames per person) but not yet wide. Expanding the subject pool would improve the generalization of models trained on DIOR.
  • Manual Initialization: While the pipeline is "semi-automated," it still requires an initial manual step to identify 10-20 joint pairs to calibrate the cameras at the start of a session.
  • Activity Variety: The current dataset focuses on walking and running; future iterations could include more complex activities or "mimicry" (people trying to hide their gait).

The authors have committed to making the pipeline open-source, providing a blueprint for other researchers to generate high-quality gait labels in difficult environments.

Summary updated

Abstract

Recently, there has been growing interest in the identification and re-identification of individuals from long distances using rooftop cameras, UAV cameras, street cams, and similar devices. This type of recognition extends beyond facial recognition, utilizing whole-body markers such as gait. However, datasets to train and test such recognition algorithms are scarce and often lack labeling. This paper introduces DIOR—a comprehensive framework for data collection, semi-automated annotation, and a dataset comprising 1.649 million RGB frames labeled with 3D/2D skeleton gait markers across 14 subjects. This dataset includes 200,000 RGB frames captured from long-range cam-eras. Our approach employs advanced 3D computer vision techniques to achieve pixel-level accuracy in indoor environments using motion capture systems. For outdoor, long-range environments, we eliminate the reliance on motion capture systems and implement a cost-effective, hybrid 3D computer vision and learning pipeline using only four inexpensive RGB cameras. This method successfully achieves precise skeleton labeling of distant subjects, even when their visual size is as small as 20-25 pixels within an RGB frame. We benchmark models trained on existing datasets such as CASIA-B, on our proposed dataset for the task of Gait recognition. Our pipeline and the accompanying dataset will be made publicly available following acceptance.
BibTeX
@inproceedings{chen2024dior,
  author = {Yuyang Chen and Praveen Raj Masilamani and Bhavin Jawade and Srirangaraj Setlur and Karthik Dantu},
  title = {{DIOR: Dataset for Indoor-Outdoor Reidentification Long Range 3D/2D Skeleton Gait Collection Pipeline, Semi-Automated Gait Keypoint Labeling and Baseline Evaluation Methods}},
  booktitle = {IJCB},
  year = {2024},
  doi = {10.1109/ijcb62174.2024.10744452},
}
Figure