Abstract
Cross-modal retrieval models often struggle to align specific words with corresponding image regions without explicit supervision. This paper introduces NAPReg (Nouns As Proxies Regularization), a novel regularization method that treats nouns as learnable "proxies" in a joint embedding space. By anchoring visual features to these semantic entities, NAPReg enhances the alignment between images and text. Evaluation on MS-COCO and Flickr30k demonstrates significant performance gains, achieving state-of-the-art results by providing fine-grained guidance to existing dual-encoder architectures.
The Challenge of Cross-Modal Alignment
In cross-modal retrieval tasks, such as finding an image based on a text description, models must map images and text into a shared embedding space where related pairs are proximal. While modern dual-encoder models utilize attention mechanisms to focus on specific image regions and words, they frequently lack explicit supervision for this alignment.
Standard training typically relies on a global "triplet loss," which associates a whole caption with a whole image. However, it does not explicitly teach the model that the word "dog" in a sentence should correspond to the specific "dog-shaped" pixels in the image. Without this fine-grained guidance, models struggle with complex scenes and fail to generalize semantic concepts across different samples in a dataset.
Methods: Nouns as Proxies
The core innovation of NAPReg is treating nouns—the most tangible entities in a sentence—as independent anchor points (proxies) within the embedding space. This approach bridges the gap between high-level ranking losses and low-level attention mechanisms.
The NAPReg Pipeline
- Proxy Creation: A Part-of-Speech (POS) tagger extracts nouns from training captions. Frequently occurring nouns are assigned learnable $d$-dimensional proxy vectors.
- Noun Context Vector ($N_V$): For any image-text pair, the model calculates a "noun context vector" by pooling visual and textual features based on their attention scores. This vector captures the essence of the entities present in that specific sample.
- Regularization Loss: The NAPReg loss function pushes the $N_V$ closer to the proxies of nouns present in the caption while pushing it away from all other "negative" noun proxies in the dataset.
- Integration: NAPReg is a "plug-and-play" module that can be added as a secondary objective to existing architectures like SCAN or SGRAF.
Experimental Results
The researchers evaluated NAPReg on three standard benchmarks: MS-COCO, Flickr30k, and Flickr8k. The addition of NAPReg consistently improved performance across all models:
- Flickr30k: Integrated with the SGRAF model, NAPReg achieved a 60.0% Recall@1 for text-to-image retrieval, a significant improvement over previous methods.
- MS-COCO: On the 5K test set, the method improved Recall@1 by 1.1% for text-to-image and 3.7% for image-to-text matching compared to the baseline.
- Qualitative Improvement: Visualization of attention maps demonstrated that NAPReg-trained models focus more precisely on correct objects, such as clearly distinguishing between different animals in the same frame.
Key Insights and Discussion
By focusing on nouns, the model builds a stable "global memory" of specific objects across the entire dataset. Because proxies are shared, information about a "car" in one image helps the model understand a "car" in a completely unrelated image. This forces attention weights to be semantically meaningful rather than just mathematically convenient for the loss function.
Limitations and Future Work
Despite its success, NAPReg relies on external POS taggers, making it susceptible to pre-processing errors. Furthermore, the current formulation is noun-centric, potentially overlooking the importance of verbs (actions) and adjectives (attributes) in complex queries. Future research may explore dynamic proxy generation or hierarchical proxies to handle rare words and fine-grained sub-categories.
Summary updated