Abstract
Existing event-stream-based trackers are primarily evaluated on short-term datasets. However, real-world tracking involves long-term scenarios where performance remains unclear. This paper introduces FELT, a large-scale, long-term frame-event visual object tracking dataset featuring 1,044 videos and 1.9 million RGB-event pairs. To address long-term challenges, we propose AMTTrack, an Associative Memory Augmented Tracker. AMTTrack utilizes Modern Hopfield Networks for multi-modal fusion and a dynamic associative memory update scheme to handle appearance variations. Extensive evaluations demonstrate AMTTrack's state-of-the-art performance across multiple benchmarks.
Introduction
Visual object tracking (VOT) is a fundamental computer vision task. While RGB cameras are standard, they often fail under high-speed motion or poor lighting. Event cameras, which record intensity changes asynchronously, offer high dynamic range and microsecond-level temporal resolution, making them ideal complements to RGB sensors. Despite this potential, current research suffers from a "short-term bias" in datasets and lacks effective mechanisms for long-term multi-modal fusion and memory.
The FELT Dataset: Bridging the Long-Term Gap
To advance the field, we present the FELT (Frame-Event Long-Term) dataset. It significantly surpasses previous benchmarks in scale and duration, providing a more realistic testing ground for trackers intended for long-duration applications like drone navigation or persistent surveillance.
FELT includes 60 object categories and 14 challenging attributes, such as full occlusion and extreme illumination changes. Analysis shows that many trackers performing well on short clips degrade significantly as sequences exceed 1,000 frames, highlighting the necessity of this long-term benchmark.
The AMTTrack Framework
We propose AMTTrack, a novel tracker designed to leverage associative memory for robust long-term tracking. Its architecture is built on two core innovations:
- Hopfield Retrieval Layer: Instead of standard cross-attention, AMTTrack uses Modern Hopfield Networks (MHNs) as an associative memory. This allows the model to "retrieve" relevant patterns between RGB and Event streams, facilitating superior multi-scale feature fusion.
- Associative Template Update Scheme: To combat "drift"—where a tracker loses the target over time—AMTTrack maintains dual Short-Term (ST) and Long-Term (LT) template memories. A Hopfield-based mechanism dynamically updates these templates, ensuring the tracker adapts to current appearance changes while remembering the original target features.
The framework adopts a "one-stream" design, processing RGB and Event tokens together within a unified Transformer backbone (ViT) for maximum efficiency and integration.
Experimental Results
AMTTrack sets a new state-of-the-art (SOTA) across several benchmarks:
- FELT Benchmark: Outperformed 21 baseline trackers with a Success Rate (SR) of 54.8 and a Normalized Precision (NPR) of 65.7.
- COESOT Dataset: Achieved an SR of 68.8, surpassing previous leaders like ViPT and UnTrack.
- Efficiency: Operates at 23 FPS, making it viable for near-real-time applications.
Ablation studies confirmed that the associative memory update scheme is critical for maintaining stability over long sequences.
Conclusion and Future Work
AMTTrack and the FELT dataset represent a significant step toward robust, long-term multi-modal tracking. While AMTTrack excels in success rate, future work will focus on improving bounding box precision by incorporating denser temporal cues and optimizing memory update frequencies. Both the dataset and source code are publicly available to foster further research in this domain.
Summary updated