Abstract
Visual tracking often faces a conflict between the need for online adaptation and real-time performance. This paper presents Recurrent Filter Learning (RFL), a framework that utilizes a Convolutional Long Short-Term Memory (ConvLSTM) network to dynamically generate object-specific filters. By replacing traditional online fine-tuning with a feed-forward recurrent update, RFL achieves state-of-the-art accuracy while maintaining a practical speed of 15 frames per second.
Introduction
Generic visual tracking requires following an arbitrary object throughout a video sequence given only its initial bounding box. Modern trackers typically leverage Convolutional Neural Networks (CNNs), but they often struggle with a critical trade-off. High-accuracy trackers frequently rely on online fine-tuning via stochastic gradient descent (SGD) to adapt to appearance changes caused by rotation, lighting, or occlusion. However, this process is computationally intensive and precludes real-time application. Conversely, standard recurrent architectures often lose vital spatial information by flattening 2D features into 1D vectors.
Methodology
The RFL framework addresses these challenges by learning to generate tracking filters in a purely feed-forward manner.
Architecture Overview
The system employs a dual-CNN architecture: an Exemplar-CNN (E-CNN) to encode the target's appearance and a Search-CNN (S-CNN) to process the search region in subsequent frames.
Convolutional LSTM and Filter Generation
At the heart of RFL is a Convolutional LSTM (ConvLSTM). Unlike standard LSTMs, the ConvLSTM replaces internal matrix multiplications with convolutions, allowing the network to maintain a 2D "memory" of the target. This preserves spatial structure and ensures shift-invariance. At each time step $t$, the ConvLSTM processes the exemplar features $e_t$ and its internal state to produce a dynamic object filter $f_t$. This filter is then convolved with the search image features $s_{t+1}$ to generate a response map $R_t$, where the peak indicates the target's new position.
Experimental Results
The model was trained offline on the ImageNet Video (VID) dataset to learn general tracking behaviors. Evaluation was conducted on major benchmarks including OTB100, VOT2016, and VOT2017.
Performance and Robustness
RFL demonstrates competitive performance, achieving an Area Under Curve (AUC) of 0.581 on OTB100, comparable to state-of-the-art trackers like SiamFC. Notably, the recurrent memory provides superior robustness in challenging scenarios such as fast motion and out-of-view occurrences, where the model can recover targets more effectively than non-recurrent counterparts.
Efficiency
By avoiding online back-propagation, RFL operates at 15 frames per second (fps) on a single GPU. This makes it significantly more viable for real-world deployment than trackers requiring expensive online optimization.
Discussion and Future Work
While RFL provides a robust framework for recurrent tracking, several areas for improvement remain. Currently, scale changes are handled via a manual scale pyramid; integrating scale estimation directly into the recurrent loop could improve elegance and performance. Additionally, long-term tracking remains a challenge as memory may drift over extended periods. Finally, the authors observed that sharing weights between E-CNN and S-CNN degraded performance, suggesting that encoding a target and searching a scene are distinct tasks requiring specialized feature representations.
Summary updated