Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset

Xiao Wang, Ju Huang, Shiao Wang, Chuanming Tang, Bo Jiang, Yonghong Tian

Abstract

Existing event-stream-based trackers are primarily evaluated on short-term datasets. However, real-world tracking involves long-term scenarios where performance remains unclear. This paper introduces FELT, a large-scale, long-term frame-event visual object tracking dataset featuring 1,044 videos and 1.9 million RGB-event pairs. To address long-term challenges, we propose AMTTrack, an Associative Memory Augmented Tracker. AMTTrack utilizes Modern Hopfield Networks for multi-modal fusion and a dynamic associative memory update scheme to handle appearance variations. Extensive evaluations demonstrate AMTTrack's state-of-the-art performance across multiple benchmarks.

Introduction

Visual object tracking (VOT) is a fundamental computer vision task. While RGB cameras are standard, they often fail under high-speed motion or poor lighting. Event cameras, which record intensity changes asynchronously, offer high dynamic range and microsecond-level temporal resolution, making them ideal complements to RGB sensors. Despite this potential, current research suffers from a "short-term bias" in datasets and lacks effective mechanisms for long-term multi-modal fusion and memory.

The FELT Dataset: Bridging the Long-Term Gap

To advance the field, we present the FELT (Frame-Event Long-Term) dataset. It significantly surpasses previous benchmarks in scale and duration, providing a more realistic testing ground for trackers intended for long-duration applications like drone navigation or persistent surveillance.

Comparison of FELT with existing event-based tracking datasets in terms of average frame count and release year.

FELT includes 60 object categories and 14 challenging attributes, such as full occlusion and extreme illumination changes. Analysis shows that many trackers performing well on short clips degrade significantly as sequences exceed 1,000 frames, highlighting the necessity of this long-term benchmark.

The AMTTrack Framework

We propose AMTTrack, a novel tracker designed to leverage associative memory for robust long-term tracking. Its architecture is built on two core innovations:

  1. Hopfield Retrieval Layer: Instead of standard cross-attention, AMTTrack uses Modern Hopfield Networks (MHNs) as an associative memory. This allows the model to "retrieve" relevant patterns between RGB and Event streams, facilitating superior multi-scale feature fusion.
  2. Associative Template Update Scheme: To combat "drift"—where a tracker loses the target over time—AMTTrack maintains dual Short-Term (ST) and Long-Term (LT) template memories. A Hopfield-based mechanism dynamically updates these templates, ensuring the tracker adapts to current appearance changes while remembering the original target features.
Architecture of AMTTrack featuring Associative Memory Transformer and Memory Update Scheme.

The framework adopts a "one-stream" design, processing RGB and Event tokens together within a unified Transformer backbone (ViT) for maximum efficiency and integration.

Experimental Results

AMTTrack sets a new state-of-the-art (SOTA) across several benchmarks:

  • FELT Benchmark: Outperformed 21 baseline trackers with a Success Rate (SR) of 54.8 and a Normalized Precision (NPR) of 65.7.
  • COESOT Dataset: Achieved an SR of 68.8, surpassing previous leaders like ViPT and UnTrack.
  • Efficiency: Operates at 23 FPS, making it viable for near-real-time applications.

Ablation studies confirmed that the associative memory update scheme is critical for maintaining stability over long sequences.

Conclusion and Future Work

AMTTrack and the FELT dataset represent a significant step toward robust, long-term multi-modal tracking. While AMTTrack excels in success rate, future work will focus on improving bounding box precision by incorporating denser temporal cues and optimizing memory update frequencies. Both the dataset and source code are publicly available to foster further research in this domain.

Summary updated

Abstract

Existing event stream based trackers undergo evaluation on short-term tracking datasets, however, the tracking of real-world scenarios involves long-term tracking, and the performance of existing tracking algorithms in these scenarios remains unclear. In this paper, we first propose a new long-term, large-scale frame-event visual object tracking dataset, termed FELT. It contains 1,044 long-term videos that involve 1.9 million RGB frames and event stream pairs, 60 different target objects, and 14 challenging attributes. To build a solid benchmark, we retrain and evaluate 21 baseline trackers on our dataset for future work to compare. In addition, we propose a novel Associative Memory Transformer based RGB-Event long-term visual tracker, termed AMTTrack. It follows a one-stream tracking framework and aggregates the multi-scale RGB/event template and search tokens effectively via the Hopfield retrieval layer. The framework also embodies another aspect of associative memory by maintaining dynamic template representations through an associative memory update scheme, which addresses the appearance variation in long-term tracking. Extensive experiments on FELT, FE108, VisEvent, and COESOT datasets fully validated the effectiveness of our proposed tracker. Both the dataset and source code will be released on https://github.com/Event-AHU/FELT_SOT_Benchmark
BibTeX
@misc{wang2024long,
  author = {Xiao Wang and Ju Huang and Shiao Wang and Chuanming Tang and Bo Jiang and Yonghong Tian},
  title = {{Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset}},
  year = {2024},
  eprint = {2403.05839},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.05839},
}
Figure