Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers

Zhiyu Zhu, Junhui Hou, Dapeng Oliver Wu

Abstract

Object tracking in challenging environments requires the integration of diverse sensor modalities. While standard RGB cameras provide rich texture, they often fail under high-speed motion or extreme lighting conditions. Event cameras offer high temporal resolution and dynamic range but lack the dense semantic information of RGB frames. This paper presents a novel "plug-and-play" training strategy designed to bridge the distribution gap between these modalities in Transformer-based trackers. By utilizing cross-modal mask modeling and orthogonal high-rank regularization, we force pre-trained Vision Transformers (ViTs) to establish meaningful cross-modal correlations. Our approach achieves state-of-the-art performance on major benchmarks without increasing inference-time computational overhead.

Introduction

Standard RGB-based object tracking frequently encounters limitations in real-world scenarios, such as motion blur, low light, and over-exposure. Event cameras, which record asynchronous pixel-level intensity changes, provide a complementary data stream that excels where RGB fails. However, effectively fusing these modalities remains a challenge. Most powerful Vision Transformers (ViTs) are pre-trained on massive RGB datasets, leading to a "distribution gap" where the model may ignore event data or fail to correlate it with RGB features.

Existing solutions often rely on complex, custom-built fusion architectures. In contrast, this research explores whether standard ViTs can be optimized for cross-modal tracking through intelligent training augmentations alone, maintaining the efficiency of the original architectures while significantly boosting their robustness.

Methodology

We propose two primary training techniques that can be integrated into existing one-stream or two-stream Transformer trackers.

1. Cross-modal Mask Modeling

Unlike traditional masked autoencoders that focus on reconstruction, our strategy employs a proactive masking approach. By randomly masking specific patches in one modality (e.g., RGB), the network is forced to rely on the corresponding patches in the alternative modality (e.g., Event) to maintain accurate tracking. This "push" mechanism encourages the model to actively seek and build stronger cross-modal correlations rather than relying solely on the dominant RGB features.

2. Orthogonal High-rank Regularization

Random masking can introduce training instability. To stabilize the process and deepen cross-modal interaction, we introduce a loss term based on Singular Value Decomposition (SVD). This loss encourages the cross-modal blocks of the attention matrix to be both high-rank and orthogonal.

In practical terms, this ensures that information projected from one modality to another is diverse and non-redundant. To maintain training efficiency, the SVD-based loss is applied to only one randomly selected Transformer layer during each optimization step, preventing the computational bottleneck typically associated with SVD operations.

Experimental Results

Datasets

The proposed methods were evaluated on two large-scale RGB-Event tracking benchmarks:

  • FE108: Comprising 108 sequences with 21 object types captured via a DAVIS346 sensor.
  • COESOT: A massive dataset featuring over 578,000 pairs across 90 object classes and 17 environmental attributes.

Performance Metrics

The augmentations yielded significant improvements across multiple state-of-the-art baselines, including CEUTrack and MonTrack:

  • FE108 Results: The "MonTrack-B+Ours" configuration achieved a 68.5% Representative Success Rate (RSR), a substantial improvement over the 64.3% baseline. It also reached a 96.2% Precision Rate (RPR).
  • COESOT Results: On the CEUTrack-L baseline, we observed a 2.2% increase in Success Rate (SR) and a 2.4% increase in Precision (PR).
  • Robustness in Extremes: The trackers showed marked improvement in challenging scenarios. On the COESOT dataset, precision in motion blur scenarios improved by 3.4%.
  • Efficiency: Because these techniques are applied only during training, the trackers maintain their original inference speeds (up to 75 FPS), ensuring zero test-time overhead.

Discussion and Key Insights

Our findings suggest that the internal attention matrices of standard ViTs often contain "scarce information" regarding the event domain. The orthogonal high-rank loss directly addresses this by filling these "empty" zones with meaningful interactions. The synergy between masking (which provides the motivation to learn) and regularization (which provides the depth and stability) allows the model to leverage the complementary strengths of both sensors effectively.

Limitations and Future Work

While effective, the SVD calculation remains computationally demanding during the training phase, even with our layer-sampling mitigation. Additionally, the model's performance is sensitive to mask size; masks that are too large may obscure the target entirely, while those that are too small may be treated as simple noise. Future research will explore the applicability of these orthogonal high-rank augmentations to other cross-modal tasks, such as RGB-3D point cloud fusion and vision-language modeling.

Summary updated

Abstract

This paper addresses the problem of cross-modal object tracking from RGB videos and event data. Rather than constructing a complex cross-modal fusion network, we explore the great potential of a pre-trained vision Transformer (ViT). Particularly, we delicately investigate plug-and-play training augmentations that encourage the ViT to bridge the vast distribution gap between the two modalities, enabling comprehensive cross-modal information interaction and thus enhancing its ability. Specifically, we propose a mask modeling strategy that randomly masks a specific modality of some tokens to enforce the interaction between tokens from different modalities interacting proactively. To mitigate network oscillations resulting from the masking strategy and further amplify its positive effect, we then theoretically propose an orthogonal high-rank loss to regularize the attention matrix. Extensive experiments demonstrate that our plug-and-play training augmentation techniques can significantly boost state-of-the-art one-stream and twostream trackers to a large extent in terms of both tracking precision and success rate. Our new perspective and findings will potentially bring insights to the field of leveraging powerful pre-trained ViTs to model cross-modal data. The code will be publicly available.
BibTeX
@inproceedings{zhu2023cross,
  author = {Zhiyu Zhu and Junhui Hou and Dapeng Oliver Wu},
  title = {{Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure