Self-Supervised Partial Cycle-Consistency for Multi-View Matching

Fedor Taggenbrock, Gertjan J. Burghouts, Ronald Poppe

Abstract

Multi-view object matching is a critical component of modern surveillance and tracking systems, yet its reliance on large-scale labeled datasets limits its scalability. Self-supervised cycle-consistency has emerged as a promising alternative, but standard formulations fail in real-world scenarios where camera views only partially overlap. This paper introduces a novel framework for partial cycle-consistency that utilizes pseudo-masking and diverse cycle variations to handle non-overlapping regions. Our method achieves a state-of-the-art F1-score of 67.4% on the DIVOTrack dataset, representing a 4.3 percentage point improvement over previous self-supervised approaches.

Introduction

In multi-camera environments, the task of re-identification—associating an object's identity across different viewpoints—is fundamental for consistent tracking. While deep learning models excel at this task when trained on massive, manually annotated datasets, the cost of labeling makes such models difficult to deploy in new, diverse environments.

Self-supervised learning via cycle-consistency offers a solution by leveraging the inherent structure of multi-view data. The core intuition is simple: if an object is matched from View A to View B, then to View C, and finally back to View A, the cycle should return to the original object. However, this logic breaks down in practical deployments where cameras have limited overlapping fields of view. When an object is visible in View A but not in View B, forcing a cycle creates a noisy learning signal that degrades model performance. This paper addresses this "diluted" signal by redefining cycle-consistency for partial overlap.

Methodology

The authors propose a multi-faceted approach to robust self-supervised matching:

1. Partial Cycle-Consistency Theory

Standard cycle-consistency requires the product of similarity matrices across a cycle to equal the identity matrix, implying every object has a correspondent in every view. The authors mathematically redefine this requirement: the cycle product must instead be a subset of the identity matrix. This theoretical shift allows the model to acknowledge that some objects will not have valid cycles, preventing the enforcement of incorrect matches.

2. Pseudo-Masking

To implement this theory without ground-truth labels, the framework employs "pseudo-masks." By analyzing the model's own high-confidence predictions, the system identifies which objects are likely visible across all views in a given cycle. The loss function then ignores inconsistencies for objects flagged as "absent," ensuring the network only learns from valid, high-certainty correspondences.

3. Trainable Cycle Variations

Rather than relying on a single path (e.g., $A \to B \to C \to A$), the method introduces four distinct "triplewise" cycle constructions. These variations allow the model to capture a wider array of feature inconsistencies, acting as an ensemble of internal constraints that provide a richer and more robust learning signal than a single cycle could provide.

4. Time-Divergent Scene Sampling

The training process utilizes a curriculum learning strategy known as time-divergent sampling. Initially, the model samples frames with small time gaps ($\Delta t$). As training progresses, the gap increases, forcing the feature extractor to become invariant not only to camera perspective but also to changes in pose and lighting that occur over time.

Experimental Evaluation

Dataset and Setup

The proposed method was evaluated on the DIVOTrack dataset, which consists of 9,000 frames across 10 scenes, featuring three overlapping camera views and an average of 19 people per frame. To further test robustness, the authors created synthetic versions of the dataset with reduced fields of view (80% and 60% overlap) to simulate challenging real-world sensor placements.

Performance Metrics

The framework achieved a state-of-the-art F1 score of 67.4%, outperforming the previous leading self-supervised method, MvMHAT, by 4.3 percentage points.

Key findings include:

  • Robustness to Low Overlap: Even when camera overlap was reduced to 60%, the proposed method outperformed baseline models trained on the full 100% overlap data.
  • Crowd Performance: The most significant gains were observed in dense, complex scenes. In the 'Ground' scene, characterized by high pedestrian density and limited overlap, the method saw a 9.1 point improvement in F1 score.

Discussion and Limitations

While the method significantly advances self-supervised re-identification, it remains dependent on the quality of the initial object detector. If the detector fails to provide a bounding box, the matching logic cannot proceed. Furthermore, the current model focuses exclusively on visual features, ignoring spatial context such as camera geometry or relative distances between objects, which could serve as additional priors for matching.

Conclusion

By explicitly modeling the absence of matches through partial cycle-consistency and pseudo-masking, this research provides a robust framework for self-supervised multi-view tracking. The ability to maintain high performance in low-overlap and high-density scenarios makes this approach particularly suited for real-world surveillance applications where manual labeling is infeasible.

Summary updated

BibTeX
@inproceedings{taggenbrock2025self,
  author = {Fedor Taggenbrock and Gertjan J. Burghouts and Ronald Poppe},
  title = {{Self-Supervised Partial Cycle-Consistency for Multi-View Matching}},
  booktitle = {VISIGRAPP (3): VISAPP},
  year = {2025},
  doi = {10.5220/0013080900003912},
}
Figure