Natural Language Person Retrieval

Tao Zhou, Jie YuAAAI 2017 · DOI

Abstract

Researchers have developed a specialized system for retrieving individuals from images using natural language descriptions. By integrating a modified deep learning architecture with a newly annotated version of the Cityscapes dataset, the system significantly outperforms generic object retrieval methods, offering advancements for video surveillance and human-robot interaction.

Introduction

The proliferation of surveillance data has made manual person identification increasingly difficult. While computer vision excels at general person detection, identifying a specific individual based on semantic descriptions (e.g., clothing, position, activity) remains a challenge. Existing systems often struggle with low resolution and occlusions. Furthermore, the lack of a dedicated benchmark dataset for natural language person retrieval has hindered progress in this field.

Methodology

The proposed framework bridges computer vision and natural language processing through a multi-stage pipeline:

  1. Region Proposals: A Faster R-CNN with a Region Proposal Network (RPN) generates potential bounding boxes, filtered by confidence and size to focus on high-quality targets.
  2. Feature Extraction:
    • Visual: A VGG-16 network (with Batch Normalization) extracts 1,000-dimensional feature vectors from image crops.
  3. Textual: Descriptions are processed using word embeddings and a Long Short-Term Memory (LSTM) network.
  4. Spatial: An 8-dimensional vector represents relative coordinates and bounding box size to interpret spatial phrases like "on the left."
  5. Architectural Innovation: To account for the arbitrary order in human descriptions (e.g., "red hat and blue jeans" vs. "blue jeans and red hat"), the authors implemented max pooling across all LSTM hidden states rather than relying solely on the final state. This "non-logic dependence" handling improved performance by 1.5%.
  6. Classification: Concatenated features are processed by a multilayer perceptron (MLP) to score the match between image regions and descriptions.
Overview of the person retrieval pipeline, showing region proposals, feature extraction, and final scoring.

Dataset and Evaluation

Because no specific dataset existed for this task, the authors adapted the Cityscapes dataset. They extracted "person" and "rider" segmentation masks and converted them into bounding boxes. Using Amazon Mechanical Turk (AMT), they collected natural language descriptions and attributes for individuals in boxes larger than 5,000 pixels to ensure visual clarity.

Key Results and Insights

The system was evaluated using Recall@K (REC@K) metrics. The proposed model, trained on the specialized Cityscapes annotations, achieved 48.8% REC@1 and 70.5% REC@2. This performance significantly exceeds random baselines and state-of-the-art generic retrieval models, such as those from UC Berkeley, which achieved only 37.3% REC@1 on this task.

Performance comparison (REC@1 and REC@2) between the proposed method and baselines on the Cityscapes dataset.

Key insights include:

  • Order Independence: Max pooling across LSTM states confirms that human descriptions are often non-linear and benefit from capturing "peak" information from every word.
  • Normalization Sensitivity: Batch Normalization improved visual accuracy by 2.3% but degraded LSTM performance by 8%, suggesting it is not universally beneficial across all neural architectures.
  • Spatial Context: The inclusion of spatial vectors is critical for disambiguating multiple people in a single scene.

Conclusion

While providing a strong proof-of-concept, the study is limited by its focus on urban street scenes and relatively large targets. Future work should address retrieval in crowded or indoor environments and improve performance for small, distant targets. Additionally, more explicit integration of attribute labels could further enhance retrieval accuracy.

Summary updated

BibTeX
@inproceedings{zhou2017natural,
  author = {Tao Zhou and Jie Yu},
  title = {{Natural Language Person Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.11083},
}
Figure