DUN: Dual-Path Temporal Matching Network for Natural Language-Based Vehicle Retrieval

Ziruo Sun, Xinfang Liu, Xiaopeng Bi, Xiushan Nie, Yilong YinWorkshops @ CVPR 2021 · OpenReview

Abstract: Retrieving vehicles matching natural language descriptions from collections of videos is a novel and uniquely challenging task, requiring consideration not only of vehicle types and colors, but also of temporal relations, e.g., "A white crossover keeping straight behind a silver hatchback." To perform this task, we propose Dual-path Temporal Matching Network (DUN). DUN uses a pre-trained CNN and GloVe to extract visual and text features, respectively, and GRUs to mine temporal relationships in videos and sentences. Furthermore, the proposed network can attain superior performance by including techniques such as re-ranking. With its simple structure, DUN achieved second place on the AI City Challenge 2021 Track 5.

Introduction

As urban environments deploy thousands of surveillance cameras, the manual search for specific vehicles based on witness descriptions—such as a "white SUV heading straight followed by a red car"—has become an insurmountable task for human operators. While traditional vehicle re-identification (Re-ID) focuses on image-to-image matching, Vehicle Retrieval by Natural Language (VR-NL) offers a more intuitive interface for users.

This task presents two primary challenges: multi-modal alignment, requiring the mapping of visual pixels and text tokens into a shared semantic space, and temporal dynamics, as descriptions often involve actions or sequences that standard static image classifiers cannot capture.

Methodology

The proposed Dual-path Temporal Matching Network (DUN) addresses these challenges by processing video and text through two parallel pipelines designed to extract and then align temporal features.

The DUN architecture featuring parallel visual and text paths with GRUs for temporal modeling.

The Visual Path

Rather than relying on generic ImageNet-trained models, DUN utilizes a ResNet101-IBN backbone pre-trained specifically on vehicle Re-ID data. This ensures the model is sensitive to fine-grained vehicle characteristics like make, model, and specific modifications. Features are extracted from cropped vehicle images across video frames using Generalized Mean (GeM) pooling.

The Text Path

Query sentences are processed into word embeddings using GloVe. This provides a robust initial representation of the linguistic components of the search query.

Temporal Matching and Loss

The core innovation lies in the use of Bidirectional Gated Recurrent Units (Bi-GRUs) in both paths. This allows the model to "mine" temporal relationships between consecutive video frames and the contextual flow of words in a sentence. To optimize the shared embedding space, the authors adapted Circle Loss for cross-modal retrieval, effectively widening the gap between matching and non-matching video-text pairs.

Datasets and Training

The model was primarily trained and evaluated on the CityFlow-NL dataset, which includes 3,028 vehicle tracks annotated with natural language descriptions. To enhance the visual feature extractor, the team incorporated CityFlowV2-ReID (real-world traffic data) and VehicleX, a synthetic dataset of 192,150 images generated by a 3D engine to provide diverse orientations and lighting conditions.

Results and Performance

The DUN model demonstrated significant improvements over the competition baseline. The integration of k-reciprocal re-ranking and model ensembling proved vital for achieving peak performance.

Performance comparison of DUN variants against the baseline across MRR and Recall metrics.

As shown in the results, the standalone DUN model reached an MRR of 0.1292, a massive jump from the baseline's 0.0269. With the addition of ensembling and re-ranking (RK), the final system achieved an MRR of 0.1613 and a Recall@10 of 0.3925. These results secured second place in the 2021 AI City Challenge Track 5.

Conclusion

DUN provides a streamlined yet effective architecture for vehicle retrieval. By prioritizing vehicle-specific pre-training and explicit temporal modeling through GRUs, it overcomes the limitations of static image-text matching. Future work may explore the integration of 3D convolutional networks or transformer-based language models like BERT to further refine temporal and semantic understanding, provided the computational costs can be managed for real-world surveillance applications.

Summary updated

Abstract

Retrieving vehicles matching natural language descriptions from collections of videos is a novel and uniquely challenging task, requiring consideration not only of vehicle types and colors, but also of temporal relations, e.g., "A white crossover keeping straight behind a silver hatchback." To perform this task, we propose Dual-path Temporal Matching Network (DUN). DUN uses a pre-trained CNN and GloVe to extract visual and text features, respectively, and GRUs to mine temporal relationships in videos and sentences. Furthermore, the proposed network can attain superior performance by including techniques such as re-ranking. With its simple structure, DUN achieved second place on the AI City Challenge 2021 Track 5.
BibTeX
@inproceedings{sun2021dun,
  author = {Ziruo Sun and Xinfang Liu and Xiaopeng Bi and Xiushan Nie and Yilong Yin},
  title = {{DUN: Dual-Path Temporal Matching Network for Natural Language-Based Vehicle Retrieval}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=hus0LfuDyvT},
}
Figure