NAPReg: Nouns As Proxies Regularization for Semantically Aware Cross-Modal Embeddings

Bhavin Jawade, Deen Dayal Mohan, Naji Mohamed Ali, Srirangaraj Setlur, Venu Govindaraju

Abstract

Cross-modal retrieval models often struggle to align specific words with corresponding image regions without explicit supervision. This paper introduces NAPReg (Nouns As Proxies Regularization), a novel regularization method that treats nouns as learnable "proxies" in a joint embedding space. By anchoring visual features to these semantic entities, NAPReg enhances the alignment between images and text. Evaluation on MS-COCO and Flickr30k demonstrates significant performance gains, achieving state-of-the-art results by providing fine-grained guidance to existing dual-encoder architectures.

The Challenge of Cross-Modal Alignment

In cross-modal retrieval tasks, such as finding an image based on a text description, models must map images and text into a shared embedding space where related pairs are proximal. While modern dual-encoder models utilize attention mechanisms to focus on specific image regions and words, they frequently lack explicit supervision for this alignment.

Standard training typically relies on a global "triplet loss," which associates a whole caption with a whole image. However, it does not explicitly teach the model that the word "dog" in a sentence should correspond to the specific "dog-shaped" pixels in the image. Without this fine-grained guidance, models struggle with complex scenes and fail to generalize semantic concepts across different samples in a dataset.

Conceptual overview of nouns acting as shared semantic anchors across diverse image-text pairs.

Methods: Nouns as Proxies

The core innovation of NAPReg is treating nouns—the most tangible entities in a sentence—as independent anchor points (proxies) within the embedding space. This approach bridges the gap between high-level ranking losses and low-level attention mechanisms.

The NAPReg Pipeline

  1. Proxy Creation: A Part-of-Speech (POS) tagger extracts nouns from training captions. Frequently occurring nouns are assigned learnable $d$-dimensional proxy vectors.
  2. Noun Context Vector ($N_V$): For any image-text pair, the model calculates a "noun context vector" by pooling visual and textual features based on their attention scores. This vector captures the essence of the entities present in that specific sample.
  3. Regularization Loss: The NAPReg loss function pushes the $N_V$ closer to the proxies of nouns present in the caption while pushing it away from all other "negative" noun proxies in the dataset.
  4. Integration: NAPReg is a "plug-and-play" module that can be added as a secondary objective to existing architectures like SCAN or SGRAF.
The NAPReg architecture, illustrating the generation of noun context vectors and the proxy-based regularization loss in a contextualized embedding space.

Experimental Results

The researchers evaluated NAPReg on three standard benchmarks: MS-COCO, Flickr30k, and Flickr8k. The addition of NAPReg consistently improved performance across all models:

  • Flickr30k: Integrated with the SGRAF model, NAPReg achieved a 60.0% Recall@1 for text-to-image retrieval, a significant improvement over previous methods.
  • MS-COCO: On the 5K test set, the method improved Recall@1 by 1.1% for text-to-image and 3.7% for image-to-text matching compared to the baseline.
  • Qualitative Improvement: Visualization of attention maps demonstrated that NAPReg-trained models focus more precisely on correct objects, such as clearly distinguishing between different animals in the same frame.

Key Insights and Discussion

By focusing on nouns, the model builds a stable "global memory" of specific objects across the entire dataset. Because proxies are shared, information about a "car" in one image helps the model understand a "car" in a completely unrelated image. This forces attention weights to be semantically meaningful rather than just mathematically convenient for the loss function.

Limitations and Future Work

Despite its success, NAPReg relies on external POS taggers, making it susceptible to pre-processing errors. Furthermore, the current formulation is noun-centric, potentially overlooking the importance of verbs (actions) and adjectives (attributes) in complex queries. Future research may explore dynamic proxy generation or hierarchical proxies to handle rare words and fine-grained sub-categories.

Summary updated

Abstract

Cross-modal retrieval is a fundamental vision-language task with a broad range of practical applications. Text-to-image matching is the most common form of cross-modal retrieval where, given a large database of images and a textual query, the task is to retrieve the most relevant set of images. Existing methods utilize dual encoders with an attention mechanism and a ranking loss for learning embeddings that can be used for retrieval based on cosine similarity. Despite the fact that these methods attempt to perform semantic alignment across visual regions and textual words using tailored attention mechanisms, there is no explicit supervision from the training objective to enforce such alignment. To address this, we propose NAPReg, a novel regularization formulation that projects high-level semantic entities i.e Nouns into the embedding space as shared learnable proxies. We show that using such a formulation allows the attention mechanism to learn better word-region alignment while also utilizing region information from other samples to build a more generalized latent representation for semantic concepts. Experiments on three benchmark datasets i.e. MS-COCO, Flickr30k and Flickr8k demonstrate that our method achieves state-of-the-art results in cross-modal metric learning for text-image and image-text retrieval tasks. Code: https://github.com/bhavinjawade/NAPReq
BibTeX
@inproceedings{jawade2023napreg,
  author = {Bhavin Jawade and Deen Dayal Mohan and Naji Mohamed Ali and Srirangaraj Setlur and Venu Govindaraju},
  title = {{NAPReg: Nouns As Proxies Regularization for Semantically Aware Cross-Modal Embeddings}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00119},
}
Figure