Abstract
The Text-to-Image Encoder-Decoder (TIED) model addresses the challenge of retrieving specific vehicle tracks from traffic camera footage using natural language queries. By integrating a shared latent space for visual and textual data with a cycle-consistent decoder that reconstructs original text descriptions from visual features, the model significantly bridges the modality gap. TIED achieved a six-fold improvement over baseline methods, ranking 7th in the 2021 AI City Challenge.
Introduction
In modern smart city applications, monitoring vehicle patterns requires searching through massive volumes of traffic camera footage. Manual searching is impractical, and traditional image retrieval systems often fail to capture the nuance of complex human descriptions, such as "A red cargo truck pulls a yellow cement mixer."
The core challenge lies in the modality gap: images and text are fundamentally different data types. To match them effectively, a system must translate both into a shared mathematical language (a latent space). Furthermore, vehicle tracks involve temporal elements and specific semantic attributes—like color and vehicle type—that are often lost in generic retrieval models.
Proposed Architecture
The authors propose the TIED architecture, which moves beyond simple encoder-only matching by introducing a reconstruction phase. The model consists of three primary components:
- Dual Encoders: A ResNet50-IBN-a backbone extracts features from vehicle image crops, while a BERT-based encoder processes natural language queries. Both streams are mapped into a shared 512-dimensional latent space.
- Joint BERT Decoder: This is the "Cycle Consistent" innovation. The decoder attempts to reconstruct the original text query using embeddings from both the image and text branches. This forces visual features to align closely with the semantic meaning of the words.
- Multi-Task Learning Objective: The model is trained using a combination of four losses: Inter-modal Triplet Loss (matching pairs), Intra-modal Triplet Loss (regularization), Attribute Loss (predicting color/type), and Cycle Loss (reconstruction error).
Dataset and Training
The study utilizes the CityFlow-NL dataset from the 2021 NVIDIA AI City Challenge, comprising 2,498 training tracks and 530 test tracks. Each track includes a sequence of vehicle images and three natural language descriptions detailing appearance and maneuvers (e.g., "turns right," "goes straight").
To handle the ambiguity of natural language (e.g., "silver" vs. "gray"), the authors implemented a semi-automated lemmatization process to create 18 color and 15 vehicle type categories. Using these as explicit training targets via the Attribute Loss significantly boosted the model's ability to anchor visual features to specific semantic concepts.
Results and Discussion
The TIED model was evaluated using Mean Reciprocal Rank (MRR). It achieved an MRR of 15.48 on the private test set, nearly six times higher than the baseline MRR of 2.69. This performance placed the team 7th overall in the 2021 AI City Challenge.
Qualitative analysis demonstrates that the cycle-consistency decoder forces the model to learn a much richer representation. While the baseline often retrieves vehicles of the correct color but wrong type, TIED's reconstruction requirement ensures that specific details—like a cement mixer being pulled by a truck—are captured in the visual embedding.
Conclusion
While TIED represents a significant leap in text-to-image retrieval for traffic monitoring, limitations remain. The current model primarily processes individual frames; explicitly modeling the temporal dimension for maneuvers like "turning left" is a priority for future work. Additionally, the authors noted sensitivity to data distribution, suggesting that more sophisticated "soft" labeling techniques could further improve attribute consensus and model robustness.
Summary updated