CALIBFREE: Self-Supervised Feature Disentanglement for Calibration-Free Multi-Camera Multi-Object Tracking

Ruiqi Xian, Iain Melvin, Deep Anil Patel, Sanjoy Kundu, Martin Renqiang Min, Dinesh ManochaSubmission to ICLR 2026 · Withdrawn

TL;DR

CALIBFREE introduces a self-supervised representation learning framework for Multi-Camera Multi-Object Tracking (MCMOT) that operates without camera calibration or manual labels. By separating features into view-specific details (for tracking across time in a single view) and view-agnostic cues (for matching across different cameras) via masked autoencoding and teacher distillation, it achieves state-of-the-art tracking accuracy while being fully resilient to camera repositioning.

What Problem Does This Paper Address?

Tracking multiple people across a network of overlapping cameras (MCMOT) is essential for surveillance, smart cities, and autonomous systems. However, most existing frameworks rely heavily on two costly prerequisites:

  1. Precise Camera Calibration: Methods often assume known camera intrinsics, extrinsics, or ground homographies. A slight physical shift or camera angle adjustment breaks calibration, requiring tedious recalibration.
  2. Extensive Human Annotations: Training cross-view re-identification models usually requires thousands of manually labeled bounding boxes and matching identities.

When moving to new environments or reconfiguring camera setups, these dependencies create severe computational and operational bottlenecks. CALIBFREE eliminates both requirements by learning robust multi-camera representations in an entirely self-supervised, calibration-free manner.

Methods

CALIBFREE builds on Masked Autoencoders (MAE) and feature disentanglement to create two distinct feature representations per detection:

                  ┌─► View-Specific (fs) ──► Distillation Decoder (ImageNet Teacher MAE)
Input Detection ──┤
(Masked Patches)  └─► View-Agnostic (fa) ──► Cross-View Attention ──► Reconstruction Decoder
  • Single-View Encoder: Uses a Vision Transformer (ViT-B) to encode unmasked patches (25% visible, 75% masked) from cropped person detections. The output token representations are split evenly into view-agnostic features ($f_a$) and view-specific features ($f_s$).
  • Single-View Knowledge Distillation: View-specific features ($f_s$) are fed into a shallow ViT decoder, trained via a Smooth L1 loss against an off-the-shelf, self-supervised ViT-L teacher (pretrained on ImageNet-1K). The teacher guides the student to capture camera-dependent textures, colors, and local details.
  • Cross-View Semantic Reconstruction: View-agnostic features ($f_a$) are pooled per detection and processed across all camera views with a shallow cross-view ViT encoder. Applying multi-head self-attention across camera views extracts viewpoint-invariant signals (e.g., pose, silhouette, aspect ratio). The resulting features are combined with $f_s$ in a reconstruction decoder to predict the masked pixel values across views via MSE loss.
  • Disentanglement Loss: A Normalized Mutual Information (NMI) loss minimizes shared information between $f_a$ and $f_s$.
  • Calibration-Free Inference: Only the single-view ViT-B encoder is evaluated at inference time. Intra-camera tracking uses $f_s$ within DeepSORT (with Kalman filtering), while cross-camera matching pairs detections directly via similarity matrices computed on $f_a$, requiring no geometric constraints or camera parameters.

Datasets

The authors evaluated CALIBFREE on two overlapping multi-camera benchmarks:

  • MMP-MvMHAT: Adapted from the MMPTRACK benchmark, featuring 4–6 overlapping indoor cameras across crowded office scenes (28 identities; 8,000 training and 4,000 validation frames) with significant occlusion and no calibration data.
  • MvMHAT: A large-scale benchmark comprising 26 video groups (98 sequences; >90,000 frames) compiled from Campus, EPFL, and newly collected footage. It contains 3–4 synchronized views with wide camera baselines (often 90° angle differences) across both indoor and outdoor environments.

Key Insights and Ideas

  • Disentangling Representation by Tracking Role: Cross-camera matching requires invariance to lighting and perspective (view-agnostic), whereas single-camera tracking benefits from fine-grained appearance details and textures (view-specific). Splitting the latent space explicitly prevents these competing objectives from interfering.
  • Pretrained MAE as a "Local Detail" Regularizer: ImageNet-pretrained MAE encoders naturally emphasize viewpoint-dependent visual cues (color, fine textures). Using one as a distillation teacher provides strong supervision for intra-camera tracking without leaking view-dependent bias into the cross-view matching branch.
  • Asymmetric Architecture Between Training and Deployment: Cross-view ViT attention is used exclusively during pretraining to force the view-agnostic representation to model shared semantics across perspectives. At test time, each camera frame is processed independently by the single-view encoder, enabling scalable, distributed deployment.

Key Results

  • Indoor Crowded Tracking (MMP-MvMHAT):
  • Reached 70.8% overall accuracy ($A$) and 66.4% overall F1 ($F$), outperforming the prior state-of-the-art self-supervised method (MvMHAT++) by 3.0% in accuracy and 7.5% in F1.
  • Achieved an over-time tracking IDF1 of 80.0 and a cross-view AIDF1 of 52.8, surpassing both supervised (SVT, DeepCC) and self-supervised baselines.
  • Resilience to Noisy Detections: When replacing ground-truth boxes with off-the-shelf YOLOX detections, CALIBFREE's performance remained remarkably steady ($F = 64.7$ vs. $66.4$), whereas baseline methods experienced severe degradation.
  • Wide-Baseline Tracking (MvMHAT): Achieved an overall accuracy of 58.4% and F1 of 58.1%, demonstrating competitive temporal and cross-view tracking even across extreme 90° viewpoint shifts.
  • Efficiency: Achieved an inference speed of 154 ms per timestep across 4 cameras (1080p resolution) on a single NVIDIA A100 GPU.

Limitations and Future Work

  • RGB-Only Associations: CALIBFREE relies exclusively on visual RGB representations and discards spatial/geometric scene relationships across views.
  • Unsupervised Geometry Integration: The authors note that developing self-supervised mechanisms to incorporate geometric and spatial relationships across detections—without introducing explicit camera calibration parameters—remains an important direction for future work.

Summary updated

Abstract

Multi-camera multi-object tracking (MCMOT) faces significant challenges in maintaining consistent object identities across varying camera perspectives, particularly when precise calibration and extensive annotations are required. In this paper, we present CALIBFREE, a self-supervised representation learning framework that does not need any calibration or manual labeling for the MCMOT task. By disentangling view-agnostic and view-specific features through single-view distillation and cross-view reconstruction, our method adapts to complex, dynamic scenarios with minimal overhead. Experiments on the MMP-MvMHAT dataset show a 3% improvement in overall accuracy and a 7. 5% increase in the average F1 score over state-of-the-art approaches, confirming the effectiveness of our calibration-free design. Moreover, on the more diverse MvMHAT dataset, our approach demonstrates superior over-time tracking and strong cross-view performance, highlighting its adaptability to a wide range of camera configurations.
BibTeX
@misc{xian2026calibfree,
  author = {Ruiqi Xian and Iain Melvin and Deep Anil Patel and Sanjoy Kundu and Martin Renqiang Min and Dinesh Manocha},
  title = {{CALIBFREE: Self-Supervised Feature Disentanglement for Calibration-Free Multi-Camera Multi-Object Tracking}},
  year = {2026},
  eprint = {2605.09245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.09245},
}

Loading related content...

Figure