Abstract
Autonomous vehicles rely on an accurate perception module. One of the fundamental challenges is to efficiently track pedestrians surrounding a vehicle to anticipate risky situations. Over the past decades, researchers have formulated the tracking problem as a data association one where they proposed various representations aiming for invariance to nuisances such as viewpoint changes, body deformation, object occlusion, and illumination changes. However, these methods still suffer to address abrupt changes since they do not explicitly model the nature of the nuisances. In this work, we propose to train a classifier that recognizes these nuisances, more specifically rotational body deformation of pedestrians. We aim to detect deformations as a method to find a good representation that will lead to better tracking of pedestrians as well as other tasks.
Rethinking "Nuisances" in Pedestrian Tracking
For autonomous vehicles (AVs), tracking pedestrians is not merely about localization; it is about predicting future intent and trajectory. Traditionally, tracking has been approached as a "data association" problem. To solve this, researchers typically strive to develop image representations that are invariant to "nuisances"—factors like changes in lighting, viewpoint, or body rotation that alter a person's visual appearance without changing their identity.
However, this research argues that attempting to ignore these nuisances is a fundamental mistake, particularly when changes are abrupt. If a pedestrian suddenly rotates, an invariant model might fail to maintain the track because the visual transformation is too extreme. This paper proposes a paradigm shift: instead of filtering out these nuisances, perception models should be trained to explicitly recognize and label them as valuable state information.
Methodology: Explicitly Modeling Rotation
The authors adapted the GOTURN (Generic Object Tracking Using Regression Networks) architecture, utilizing a CaffeNet backbone to extract high-level features. The system processes two cropped images—a "Search Image" and a "Target Image"—through five convolutional layers. The resulting features are concatenated and passed through three fully connected layers.
Unlike standard trackers, this model acts as a multi-class classifier. It attempts to determine two critical pieces of information:
- Identity Re-identification: Is the person in both frames the same individual?
- Rotational Displacement: What is the specific rotation angle between the two frames, classified into eight 45-degree increments?
Dataset Construction and the Labeling Challenge
Because existing datasets lacked specific rotation labels, the authors synthesized a new dataset by processing the IAS-Lab RGBD-ID and BIWI RGBD-ID datasets. While these datasets provided skeletal information and person IDs, they did not explicitly label orientation.
The authors used the positions of the left and right shoulders to calculate a quaternion representing body orientation. However, skeletal data alone is ambiguous; shoulder alignment is identical whether a person is facing toward or away from the camera. To resolve this "0° vs. 180° problem," the team integrated an OpenCV face detector to determine if the person was facing the camera, enabling accurate labeling across the full 360-degree spectrum. The final processed dataset comprised over 700,000 image pairs.
Results and the Generalization Gap
The model was evaluated under two distinct scenarios to test its robustness:
- Scenario 1 (Shared IDs/Backgrounds): When tested on data containing the same backgrounds and individuals seen during training, the model achieved a remarkable 98.5% accuracy in detecting both rotation and identity.
- Scenario 2 (Unseen Data): When moved to entirely new environments and subjects, performance dropped significantly. Combined accuracy fell to 30.8%, although identity detection alone remained more resilient at 60.8%.
These results highlight a significant "generalization gap." While the architecture is capable of learning rotation, it is highly susceptible to overfitting, likely memorizing environmental "noise" rather than the abstract concept of human rotation.
Future Directions
The primary challenge moving forward is improving generalization. The authors propose several strategies to mitigate overfitting:
- Advanced Data Augmentation: Implementing jitter, noise, and "image building" (compositing pedestrian crops onto diverse backgrounds) to force the network to focus on the subject rather than the scene.
- Expanding Nuisance Modeling: Extending the framework to include other nuisances such as partial occlusions and illumination changes.
- Real-world Deployment: Integrating the nuisance-aware tracker into the Loomo robot platform to evaluate performance in live, dynamic navigation scenarios.
Summary updated