Abstract
Current event-/frame-event based trackers undergo evaluation on short-term tracking datasets, however, the tracking of real-world scenarios involves long-term tracking, and the performance of existing tracking algorithms in these scenarios remains unclear. In this paper, we first propose a new long-term and large-scale frame-event single object tracking dataset, termed FELT. It contains 1,044 videos and 1,949,680 RGB frames and event stream pairs and has become the largest frame-event tracking dataset to date. We re-train and evaluate 15 baseline trackers on our dataset for future works to compare. More importantly, we find that the RGB frames and event streams are naturally incomplete due to the influence of challenging factors and spatially sparse event flow. In response to this, we propose a novel associative memory Transformer network as a unified backbone by introducing modern Hopfield layers into multi-head self-attention blocks to fuse both RGB and event data. Extensive experiments on RGB-Event (FELT), RGB-Thermal (RGBT234, LasHeR), and RGB-Depth (DepthTrack) datasets fully validated the effectiveness of our model.
Introduction
Visual object tracking typically relies on standard RGB cameras, which often struggle in "edge case" scenarios such as high-speed motion (causing motion blur) or extreme lighting (low light or high dynamic range). Event cameras address these limitations by recording asynchronous intensity changes as "spikes," offering high temporal resolution and low power consumption. However, event cameras are inherently blind to static objects, making a hybrid RGB-Event approach the ideal solution for robust tracking.
Despite the potential of hybrid tracking, two major gaps remain. First, existing RGB-Event datasets are primarily short-term, failing to represent real-world scenarios like drone navigation or surveillance where targets may disappear or change appearance over long durations. Second, current tracking algorithms often suffer from "model drift" during long sequences because they lack a robust mechanism to maintain a long-term memory of the target's appearance.
The FELT Dataset
To address the data gap, we introduce FELT (Frame-Event Long-Term), the largest long-term RGB-Event tracking dataset to date. FELT comprises 1,044 videos and over 1.9 million RGB/Event pairs, featuring 60 object classes and 14 challenging attributes, including fast motion, full occlusion, and out-of-view scenarios.
As shown in the figure above, FELT significantly exceeds previous benchmarks like FE108, VisEvent, and COESOT in both sequence length and total scale, providing a more rigorous testing ground for long-term tracking stability.
AMTTrack: Associative Memory Augmented Tracker
To set a high performance standard on this new benchmark, we developed AMTTrack (Associative Memory Augmented Tracker). The core of AMTTrack is a one-stream Vision Transformer (ViT) backbone enhanced by Modern Hopfield Networks (MHN), which act as an "associative memory" for multi-modal fusion and template management.
Key Methodological Innovations:
- Cross-Modal Retrieval: Instead of simple feature concatenation, AMTTrack uses a Hopfield retrieval layer. This allows RGB search tokens to query Event template tokens (and vice versa), effectively using one modality to "fill in the blanks" of the other—for instance, using event data to resolve a motion-blurred RGB frame.
- Associative Template Update: To handle long-term tracking, the model maintains a memory bank of both short-term (recent) and long-term (diverse) templates. A Hopfield-based update scheme ensures the tracker retains a robust representation of the object even through significant appearance shifts or reappearances after occlusion.
Performance and Insights
AMTTrack demonstrates state-of-the-art performance across multiple benchmarks:
- FELT Benchmark: AMTTrack outperformed 21 baseline trackers, achieving a Success Rate (SR) of 54.8% and a Normalized Precision (NPR) of 65.7%.
- FE108 Dataset: On this established benchmark, it achieved a Precision (PR) of 95.9%, surpassing previous leaders like MamTrack and ViPT.
- Efficiency: Despite its sophisticated memory architecture, the tracker maintains a real-time speed of 23 FPS.
Our analysis confirms that treating memory as a continuous retrieval process is more robust than standard linear fusion. By using RGB as a spatial anchor and Events as a motion anchor, AMTTrack successfully mitigates the "static blindness" of event sensors while exploiting their high-speed advantages.
Conclusion and Future Work
While AMTTrack is highly effective at target retrieval, there is room to improve localization precision, as its bounding box tightness is slightly lower than some temporal-only models. Future work will focus on integrating more explicit frame-to-frame temporal cues and exploring denser temporal resolutions within event data to further leverage the unique properties of event-based sensors.
Summary updated