Optimizing Long-Term Robot Tracking with Multi-Platform Sensor Fusion

Giuliano Albanese, Arka Mitra, Jan-Nico Zaech, Yupeng Zhao, Ajad Chhatkuli, Luc Van Gool

Abstract

Tracking visually identical agents over long durations remains a significant hurdle in multi-object tracking (MOT). In the RoboCup Standard Platform League (SPL), humanoid robots are hardware-identical, and jersey numbers are often too small for reliable detection. This paper introduces a robust framework that addresses these challenges by fusing external video data with the robots' internal sensor logs. By formulating the tracking problem as a global quadratic binary optimization, the system matches visual "tracklets" to internal state estimates. Experimental results on real-world tournament data demonstrate an 88.11% identification recall, doubling the performance of state-of-the-art visual-only baselines.

Introduction

Multi-object tracking in sports typically relies on unique visual features—such as jersey patterns or player physique—to maintain identity through occlusions. However, the RoboCup SPL environment presents a unique set of difficulties:

  1. Visual Indistinguishability: Robots on the same team are identical, rendering standard Re-Identification (Re-ID) methods ineffective.
  2. Severe Occlusions: Clusters of robots and human referees frequently obstruct the camera's view.
  3. Uncalibrated Wide-Angle Video: Recordings often suffer from significant barrel distortion from stationary, wide-angle lenses.

To overcome these, the authors propose leveraging the robots' internal "heartbeat" data—specifically their self-localization estimates—as a temporal signature to disambiguate identities.

Methodology

The proposed pipeline treats tracking as a data fusion problem, integrating visual evidence with internal robot states through several stages.

The multi-stage pipeline for robot tracking, from raw video and game logs to global optimization and final tracks.

1. Camera Calibration and Tracklet Generation

The system first rectifies wide-angle video by using the known geometry of the soccer field to estimate camera parameters. Short, reliable visual "tracklets" are then generated using Tracktor, a method based on Faster R-CNN that maintains local continuity.

2. Global Optimization

The core of the framework is a constrained quadratic binary optimization problem. This global approach ensures that identity assignments are consistent across the entire sequence rather than making greedy, frame-by-frame decisions. The objective function minimizes costs associated with:

  • Self-localization: Matching visual positions to the $(x, y)$ coordinates reported by the robots' internal sensors.
  • Jersey Color: Using a VGG16-based classifier to distinguish between Home and Away teams.
  • Fallen State: Correlating visual instances of a robot lying down with internal IMU data.
  • Trajectory Continuity: Bridging gaps between tracklets using a constant velocity model.

Weighting for these cost terms is automatically tuned using Particle Swarm Optimization (PSO) to account for varying noise levels across different matches.

Key Insights: Internal Sensors as "Soft" Biometrics

The most powerful feature for disambiguation is the robot's own belief of its position on the field. Even when noisy, these self-localization trajectories provide a unique temporal signature that visual data lacks.

Visualization of tracklet trajectories (red) being matched to internal self-localization trajectories (dotted) on the field coordinates.

An ablation study highlighted the criticality of this fusion: removing self-localization data caused the Mean Player Identification Recall (MPIR) to drop from 88% to just 15%. This confirms that in environments with identical agents, internal state data is essential for long-term tracking.

Results and Evaluation

The framework was evaluated on 8 annotated sequences from the 2019 and 2022 RoboCup World Cups. The system was compared against an "Oracle" version of DeepSORT, which was manually initialized with correct IDs.

  • Proposed Fusion Method: 88.11% MPIR
  • Oracle DeepSORT: 43.76% MPIR

The proposed method's ability to recover identities after long occlusions or complex interactions allowed it to significantly outperform the visual-only baseline.

Limitations and Future Work

While highly effective for post-game analysis, the current system assumes a stationary camera and does not yet interpolate paths during full occlusions. Future research will focus on adapting the calibration for moving cameras and optimizing the solver for real-time applications, such as live broadcast enhancement or autonomous coaching tools.

Summary updated

Abstract

Monitoring a fleet of robots requires stable long-term tracking with re-identification, which is yet an unsolved challenge in many scenarios. One application of this is the analysis of autonomous robotic soccer games at RoboCup. Tracking in these games requires handling of identically looking players, strong occlusions, and non-professional video recordings, but also offers state information estimated by the robots. In order to make effective use of the information coming from the robot sensors, we propose a robust tracking and identification pipeline. It fuses external non-calibrated camera data with the robots’ internal states using quadratic optimization for tracklet matching. The approach is validated using game recordings from previous RoboCup World Cup tournaments.
BibTeX
@inproceedings{albanese2024optimizing,
  author = {Giuliano Albanese and Arka Mitra and Jan-Nico Zaech and Yupeng Zhao and Ajad Chhatkuli and Luc Van Gool},
  title = {{Optimizing Long-Term Robot Tracking with Multi-Platform Sensor Fusion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00684},
}
Figure