PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

Yuan Yao, Qianyu Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

Abstract

Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mainstream due to their superior computation efficiency and competitive performance. However, the removal of object detectors also deprives the capability of VLP models in explicit object modeling, which is essential to various position-sensitive vision-language (VL) tasks, such as referring expression comprehension and visual commonsense reasoning. To address the challenge, we introduce PEVL that enhances the pre-training and prompt tuning of VLP models with explicit object position modeling. Specifically, PEVL reformulates discretized object positions and language in a unified language modeling framework, which facilitates explicit VL alignment during pre-training, and also enables flexible prompt tuning for various downstream tasks. We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs. We make the data and code for this paper publicly available at https://github.com/thunlp/PEVL.

Introduction

In Vision-Language Pre-training (VLP), a significant shift has occurred from detector-based models (e.g., Faster R-CNN) to "detector-free" architectures that process raw image patches. While detector-free models offer superior computational efficiency, they often lack the explicit spatial awareness required for position-sensitive tasks. Tasks such as Referring Expression Comprehension (REC), Phrase Grounding, and Visual Commonsense Reasoning (VCR) depend heavily on the model's ability to link specific linguistic entities to precise spatial coordinates. PEVL (Position-enhanced Vision-Language) addresses this gap by integrating explicit object position modeling into a unified language modeling framework without re-introducing the overhead of traditional object detectors.

Methodology

PEVL introduces a novel approach by treating spatial positions as a part of the linguistic vocabulary. This unification allows the model to leverage standard Transformer architectures for both visual and spatial reasoning.

Position Discretization and GMLM

Instead of treating bounding boxes as continuous coordinates, PEVL discretizes them into integer tokens. A bounding box defined by $(x_{min}, y_{min}, x_{max}, y_{max})$ is converted into four discrete tokens (e.g., [310] [73] [406] [475]) and added to the model's vocabulary.

During pre-training, the model employs Generalized Masked Language Modeling (GMLM). Unlike standard MLM which only masks words, GMLM masks these position tokens. This forces the model to reconstruct spatial coordinates based on visual features and surrounding text, effectively learning fine-grained vision-language alignment.

Overview of PEVL pre-training and downstream prompt tuning for various vision-language tasks.

Ordering-aware Objective and Prompt Tuning

To account for the geometric nature of coordinates, PEVL uses an ordering-aware objective. Rather than standard cross-entropy, it employs a soft-labeling system that assigns higher probabilities to tokens representing nearby coordinates, reflecting that a "near miss" in geometry is better than a distant prediction.

For downstream adaptation, PEVL utilizes Unified Prompt Tuning. By framing tasks as "fill-in-the-blank" problems—such as The horse < [M] [M] [M] [M] > for REC—the model can reuse the pre-trained GMLM head to predict coordinates or labels directly, maintaining architectural consistency across tasks.

Experiments and Results

PEVL was pre-trained on a massive dataset of 4.7 million image-text pairs with explicit object annotations, including RefCOCO, Flickr30k Entities, and Visual Genome.

Performance Highlights

  • Referring Expression Comprehension: PEVL achieved a significant leap over its baseline (ALBEF), showing improvements of +24.5 to +28.2 absolute points on RefCOCO+. It also outperformed regression-based models like MDETR by 2.9 points on the RefCOCO+ testA set.
  • Visual Relation Detection: On the Visual Genome dataset, PEVL improved Recall@50 by 6.8 points, demonstrating superior understanding of spatial interactions.
  • Grounded Reasoning: Even for position-insensitive tasks like VQA, providing grounded inputs (explicit object locations) led to performance gains, suggesting that spatial awareness is a fundamental component of visual logic.

Conclusion and Limitations

PEVL successfully bridges the gap between efficient detector-free models and the precision of spatial modeling. By discretizing positions into the language space, it achieves state-of-the-art results across various benchmarks.

However, challenges remain. The addition of position tokens increases sequence length, impacting inference speed in dense scenes. Furthermore, while the model is detector-free at inference, it still relies on annotated bounding boxes during pre-training. Future work may explore weakly supervised methods to learn spatial positions from raw image-text data without manual box annotations.

Summary updated

Abstract

Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mainstream due to their superior computation efficiency and competitive performance. However, the removal of object detectors also deprives the capability of VLP models in explicit object modeling, which is essential to various position-sensitive vision-language (VL) tasks, such as referring expression comprehension and visual commonsense reasoning. To address the challenge, we introduce PEVL that enhances the pre-training and prompt tuning of VLP models with explicit object position modeling. Specifically, PEVL reformulates discretized object positions and language in a unified language modeling framework, which facilitates explicit VL alignment during pre-training, and also enables flexible prompt tuning for various downstream tasks. We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs. We make the data and code for this paper publicly available at https://github.com/thunlp/PEVL.
BibTeX
@inproceedings{yao2022pevl,
  author = {Yuan Yao and Qianyu Chen and Ao Zhang and Wei Ji and Zhiyuan Liu and Tat-Seng Chua and Maosong Sun},
  title = {{PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.763},
}
Figure