Abstract
Tracking visually identical agents over long durations remains a significant hurdle in multi-object tracking (MOT). In the RoboCup Standard Platform League (SPL), humanoid robots are hardware-identical, and jersey numbers are often too small for reliable detection. This paper introduces a robust framework that addresses these challenges by fusing external video data with the robots' internal sensor logs. By formulating the tracking problem as a global quadratic binary optimization, the system matches visual "tracklets" to internal state estimates. Experimental results on real-world tournament data demonstrate an 88.11% identification recall, doubling the performance of state-of-the-art visual-only baselines.
Introduction
Multi-object tracking in sports typically relies on unique visual features—such as jersey patterns or player physique—to maintain identity through occlusions. However, the RoboCup SPL environment presents a unique set of difficulties:
- Visual Indistinguishability: Robots on the same team are identical, rendering standard Re-Identification (Re-ID) methods ineffective.
- Severe Occlusions: Clusters of robots and human referees frequently obstruct the camera's view.
- Uncalibrated Wide-Angle Video: Recordings often suffer from significant barrel distortion from stationary, wide-angle lenses.
To overcome these, the authors propose leveraging the robots' internal "heartbeat" data—specifically their self-localization estimates—as a temporal signature to disambiguate identities.
Methodology
The proposed pipeline treats tracking as a data fusion problem, integrating visual evidence with internal robot states through several stages.
1. Camera Calibration and Tracklet Generation
The system first rectifies wide-angle video by using the known geometry of the soccer field to estimate camera parameters. Short, reliable visual "tracklets" are then generated using Tracktor, a method based on Faster R-CNN that maintains local continuity.
2. Global Optimization
The core of the framework is a constrained quadratic binary optimization problem. This global approach ensures that identity assignments are consistent across the entire sequence rather than making greedy, frame-by-frame decisions. The objective function minimizes costs associated with:
- Self-localization: Matching visual positions to the $(x, y)$ coordinates reported by the robots' internal sensors.
- Jersey Color: Using a VGG16-based classifier to distinguish between Home and Away teams.
- Fallen State: Correlating visual instances of a robot lying down with internal IMU data.
- Trajectory Continuity: Bridging gaps between tracklets using a constant velocity model.
Weighting for these cost terms is automatically tuned using Particle Swarm Optimization (PSO) to account for varying noise levels across different matches.
Key Insights: Internal Sensors as "Soft" Biometrics
The most powerful feature for disambiguation is the robot's own belief of its position on the field. Even when noisy, these self-localization trajectories provide a unique temporal signature that visual data lacks.
An ablation study highlighted the criticality of this fusion: removing self-localization data caused the Mean Player Identification Recall (MPIR) to drop from 88% to just 15%. This confirms that in environments with identical agents, internal state data is essential for long-term tracking.
Results and Evaluation
The framework was evaluated on 8 annotated sequences from the 2019 and 2022 RoboCup World Cups. The system was compared against an "Oracle" version of DeepSORT, which was manually initialized with correct IDs.
- Proposed Fusion Method: 88.11% MPIR
- Oracle DeepSORT: 43.76% MPIR
The proposed method's ability to recover identities after long occlusions or complex interactions allowed it to significantly outperform the visual-only baseline.
Limitations and Future Work
While highly effective for post-game analysis, the current system assumes a stationary camera and does not yet interpolate paths during full occlusions. Future research will focus on adapting the calibration for moving cameras and optimizing the solver for real-time applications, such as live broadcast enhancement or autonomous coaching tools.
Summary updated