CalibFree: Self-Supervised View Feature Separation for Calibration-Free Multi-Camera Multi-Object Tracking

Ruiqi Xian, Deep Patel, Iain Melvin, Sanjoy Kundu, Martin Renqiang Min, Dinesh Manocha

TL;DR

CalibFree is a fully self-supervised framework for Multi-Camera Multi-Object Tracking (MCMOT) that operates entirely without camera calibration or manual annotations. By separating object features into view-agnostic and view-specific components via single-view distillation and cross-view reconstruction, it achieves state-of-the-art tracking performance while remaining highly resilient to real-world camera drift and malfunctions.

What Problem Does This Paper Address?

Multi-Camera Multi-Object Tracking (MCMOT) is essential for maintaining consistent object identities across wide areas, minimizing blind spots, and overcoming occlusions in environments like smart cities and surveillance networks.

However, traditional MCMOT systems rely heavily on precise camera calibration (intrinsics and extrinsics) or scene-specific geometric priors to match identities across different viewpoints. In the real world, camera networks are rarely static: cameras drift over time, undergo maintenance, or experience subtle repositioning. These dynamic shifts quickly invalidate calibration, leading to tracking failures and requiring costly manual recalibration or reannotation.

Fig. 1: Multi-Camera Multi-Object Tracking (MCMOT) setup. Left: Multi- view scenes with individuals tracked across overlapping camera views, each assigned a unique color-coded bounding box. Top right: Flexible camera configurations illustrat- ing variable camera numbers and placements across scenarios, or due to factors like malfunction or relocation. Bottom right: Examples of appearance variations for indi- viduals across different viewpoints, highlighting the challenge of maintaining consistent identity association in multi-view tracking.

To solve this, the authors introduce CalibFree, a purely RGB-driven, self-supervised approach that leverages visual co-occurrences in overlapping fields of view to learn cross-view alignment without any geometric constraints or manual labels.

Methods

CalibFree introduces a novel distillation-and-reconstruction training scheme built on a shared Vision Transformer (ViT) encoder. The core pipeline consists of three main pillars:

  1. Feature Separation: Person detections are cropped, masked, and passed through the ViT encoder. The resulting features are split evenly into two components:
  2. View-agnostic features ($f_a$): Target identity-consistent cues (like silhouette, body shape, and pose) for cross-view matching.
  3. View-specific features ($f_s$): Capture fine-grained, camera-dependent appearance details (like lighting and texture) for local tracking.
  4. A Normalized Mutual Information (NMI) loss acts as a separation regularizer, forcing these two representations to share minimal information.
  5. Single-View Distillation: To stabilize tracking within an individual camera view, the view-specific features ($f_s$) are refined by distilling patch-level knowledge from a powerful, pretrained Masked Autoencoder (MAE) teacher model.
  6. Cross-View Reconstruction: To align the view-agnostic features ($f_a$) across different cameras, a shallow cross-view transformer encoder applies self-attention over all detections at a given timestamp. Because the same masking pattern is applied across all views, the model must use unmasked regions from other cameras to reconstruct the missing pixels of a person in a given view. This forces the view-agnostic embeddings of the same individual to naturally cluster together.

During inference, the view-specific features drive standard intra-camera tracking (via DeepSORT), while the view-agnostic features are used for bipartite matching to merge identities across different camera views.

Datasets

The framework was evaluated on two challenging multi-view tracking benchmarks:

  • MMP-MvMHAT: An indoor dataset featuring 4–6 overlapping cameras tracking 28 individuals in crowded, occluded environments. It provides 8,000 annotated training frames and 4,000 validation frames, notably containing no camera calibration data.
  • MvMHAT: A large-scale dataset containing 26 video groups (98 sequences) across diverse indoor and outdoor scenes with 3–4 synchronized views. It totals over 90,000 annotated frames and features extreme viewpoint variations (often up to 90 degrees).

Key Insights and Ideas

  • Visual Co-occurrence as a Natural Label: In multi-camera setups with overlapping fields of view, the simultaneous appearance of an object across views provides an implicit supervisory signal. CalibFree exploits this without needing explicit identity labels.
  • The Necessity of Feature Separation: Standard tracking models often suffer from camera-specific appearance biases (e.g., a jacket looking red in one camera but brown under another's lighting). Explicitly separating features ensures that environment-specific details don't corrupt cross-view identity matching, while still preserving them for local temporal tracking.
  • Reconstruction as an Alignment Mechanism: Forcing a network to patch together an image using data from a completely different camera perspective inherently strips away view-specific biases, leaving behind a pure, view-invariant identity representation.

Key Results

CalibFree significantly outperforms both supervised and self-supervised baselines across key tracking metrics:

  • MMP-MvMHAT Performance: CalibFree achieved a 3% improvement in overall accuracy and a 7.5% increase in the average F1 score over previous state-of-the-art approaches. It demonstrated an exceptionally high IDF1 score (80.0), reflecting strong identity continuity under heavy crowding.
  • Robustness to Noisy Detections: Unlike prior self-supervised methods that suffer severe performance drops when switched from ground-truth bounding boxes to noisy, automated detections (like YOLOX or YOLOv7), CalibFree maintained high ID association consistency with minimal degradation.
  • Resilience to Camera Dynamics: In controlled perturbation experiments simulating camera misalignment (cropping/shifting fields of view) and camera malfunctions (complete sensor failure), CalibFree remained highly robust, proving its readiness for unpredictable, real-world deployments.

Limitations and Future Work

Despite its strong performance, CalibFree relies exclusively on RGB signals, completely ignoring geometric relationships across views. While learning geometry without camera parameters is highly non-trivial, the authors acknowledge that integrating self-supervised geometric cues could further enhance identity consistency.

Future research directions will focus on exploring self-supervised strategies to incorporate these spatial relationships, allowing the model to better leverage relative positioning under dynamic camera configurations.

Summary updated

Abstract

Multi-camera multi-object tracking (MCMOT) faces significant challenges in maintaining consistent object identities across varying camera perspectives, particularly when precise calibration and extensive annotations are required. In this paper, we present CalibFree, a self-supervised representation learning framework that does not need any calibration or manual labeling for the MCMOT task. By promoting feature separation between view-agnostic and view-specific representations through single-view distillation and cross-view reconstruction, our method adapts to complex, dynamic scenarios with minimal overhead. Experiments on the MMP-MvMHAT dataset show a 3% improvement in overall accuracy and a 7.5% increase in the average F1 score over state-of-the-art approaches, confirming the effectiveness of our calibration-free design. Moreover, on the more diverse MvMHAT dataset, our approach demonstrates superior over-time tracking and strong cross-view performance, highlighting its adaptability to a wide range of camera configurations. Code will be publicly available upon acceptance.
BibTeX
@misc{xian2026calibfree,
  author = {Ruiqi Xian and Deep Patel and Iain Melvin and Sanjoy Kundu and Martin Renqiang Min and Dinesh Manocha},
  title = {{CalibFree: Self-Supervised View Feature Separation for Calibration-Free Multi-Camera Multi-Object Tracking}},
  year = {2026},
  eprint = {2605.09245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.09245},
}

Loading related content...

Figure