Abstract
Image captioning has traditionally oscillated between region-based features, which focus on detected objects, and grid-based features, which capture the entire scene. While grid features offer comprehensive coverage, their integration into Transformer architectures often sacrifices local spatial relationships due to sequence flattening. This paper introduces the Locality-Sensitive Transformer Network (LSTNet), a novel architecture designed to restore "local awareness" to grid-based visual modeling. By implementing Locality-Sensitive Attention (LSA) and Locality-Sensitive Fusion (LSF), LSTNet achieves state-of-the-art performance on the MS-COCO benchmark while maintaining significantly higher efficiency than hybrid region-grid models.
Introduction
In the field of image captioning, the visual encoder's role is to translate an image into a representation that a language decoder can process. Historically, this has been achieved through two primary methods: Region Features, which utilize bounding boxes from object detectors like Faster R-CNN, and Grid Features, which use raw feature maps from a CNN.
While region features provide clear object-level semantics, they often overlook background context and suffer from noise in the detection process. Grid features capture the full scene but present a different challenge: when fed into a Transformer, these 2D maps are flattened into 1D sequences. This process destroys the spatial proximity between adjacent grids. If a single object spans multiple grids, a standard Transformer treats them as independent tokens, making it difficult for the model to reconstruct the local structure of the object. LSTNet addresses this "fragmentation" problem by re-introducing locality into the Transformer framework.
Methodology
The LSTNet architecture introduces two specific mechanisms to enhance local modeling without sacrificing the global context-heavy nature of the Transformer.
Locality-Sensitive Attention (LSA)
LSA is designed to improve intra-layer interactions. Standard self-attention mechanisms are inherently global, looking at the relationship between every token in a sequence. LSA complements this by using a multi-branch structure that includes identity mappings, 1x1 convolutions, and 3x3 convolutions. This allows the model to perceive fine-grained local information across different receptive fields simultaneously.
To maintain efficiency, the authors employ a re-parameterization strategy. During the training phase, the multiple branches learn diverse features. For inference, these branches are mathematically collapsed into a single 3x3 convolution. This ensures that the model remains lightweight and fast during deployment while benefiting from multi-scale training.
Locality-Sensitive Fusion (LSF)
While LSA handles interactions within a single layer, LSF manages inter-layer information. In standard architectures, the decoder typically only receives the output of the final encoder layer. LSF instead aggregates features from all encoder layers, recognizing that different layers capture different levels of abstraction (e.g., low-level textures vs. high-level semantics).
To facilitate this fusion, LSTNet utilizes a Spatial Shift operation. By shifting portions of the feature maps in four directions (up, down, left, right), the model allows a grid at any given coordinate to align and interact with its neighbors during the fusion process. This provides a computationally inexpensive way to strengthen spatial correlations across the network's depth.
Experimental Results
Performance on MS-COCO
The model was primarily evaluated on the MS-COCO dataset, using the CIDEr metric as the primary benchmark for caption quality. LSTNet achieved a CIDEr score of 134.8 on the offline Karpathy test split. On the official MS-COCO online leaderboard, it reached 136.3, surpassing previous state-of-the-art models such as DLCT and RSTNet. Experiments on Flickr8k and Flickr30k further confirmed the model's robustness and generalization capabilities.
Efficiency and Qualitative Gains
One of the most significant advantages of LSTNet is its speed. During the cross-entropy training stage, LSTNet is approximately 3x faster than models like DLCT. This efficiency stems from avoiding the dual-stream processing of both region and grid features, relying instead on optimized grid modeling.
Qualitatively, attention map visualizations indicate that LSTNet is more adept at focusing on primary subjects. While vanilla Transformers may distribute attention across irrelevant background elements, LSTNet’s locality-sensitive mechanisms allow it to "lock on" to specific objects, such as cell phones or animals, leading to more accurate and descriptive captions.
Conclusion
LSTNet demonstrates that grid features can outperform region-based approaches if the inherent spatial locality of the image is preserved. By bridging the gap between the global modeling of Transformers and the local modeling of CNNs through LSA and LSF, the authors provide a high-performance, efficient alternative for image captioning. Future research may extend these locality-sensitive principles to other multi-modal tasks, such as Visual Question Answering (VQA) and video analysis, where the interplay between local detail and global context is equally vital.
Summary updated