Synonym relations affect object detection learned on vision-language data

Giacomo Nebbia, Adriana KovashkaNAACL 2024 · DOI

Introduction

Current vision-language object detectors are surprisingly brittle: swap "dog" for "canine," and performance often tanks. While modern models like BARON or OWL-ViT are designed for "open-vocabulary" detection—theoretically allowing them to detect anything described in text—they remain sensitive to lexical variation. This paper introduces two simple yet effective strategies—back-translation data augmentation and class embedding enrichment—to help models recognize objects regardless of the specific synonym used, boosting synonym detection performance by over 4% mAP.

The Synonym Gap in Object Detection

We are entering an era where detectors are trained on massive amounts of image-caption pairs rather than fixed label sets. However, human language is diverse; one person might call an object a "sofa," while another calls it a "couch." Even state-of-the-art models struggle with this variation. If a model is trained primarily on the word "person," it might fail to detect that same object when queried for "man," "woman," or "human."

Standard vision-language object detection setup where text prompts are matched against image regions.

Experimental results confirm this "lexical brittleness." A baseline model saw a massive performance drop when tested on synonyms, falling from ~44% mAP on standard names to ~33% on synonyms. This suggests that "understanding" a concept in these models is often tied too closely to specific strings of text rather than visual semantics.

Methodology

The authors propose a multi-pronged approach to bake synonym robustness into the training process.

Overview of the proposed framework, including back-translation for data augmentation and class embedding enrichment.

1. Data Augmentation by Back-Translation (BT)

To increase caption diversity without manual labeling, the authors used a "telephone game" approach. Original English captions were translated into German or Russian and then back into English. Because machine translation is not a 1:1 mapping, this process naturally introduces synonyms (e.g., "skate board rider" becomes "skateboarder"). These semantically similar captions are then added to the training set to provide linguistic variety.

2. Class Embedding Enrichment

Usually, a detector represents a class by running a prompt like "a photo of a person" through a text encoder. The authors enriched this by creating a "bag of synonyms." They averaged the embeddings of the main class name along with its synonyms and related terms (e.g., averaging vectors for "person," "man," and "woman"). For object detection, treating "man" and "woman" as synonyms for "person" is visually logical, even if they are linguistically distinct.

3. Curriculum Learning (CL)

Training heavily on enriched embeddings can sometimes slightly degrade performance on standard class names. To mitigate this, the authors implemented a curriculum: the model starts training on original, simple embeddings and gradually transitions to enriched, synonym-aware embeddings over time.

Key Results and Insights

The proposed methods yielded significant improvements in making detectors more flexible:

  • Synonym Performance: The baseline [email protected] on synonyms was 33.87%. By using the Curriculum Learning approach with enriched embeddings, this jumped to 37.93%.
  • Robustness: Back-translation alone improved synonym performance while reducing the variance in how the model handled different synonym sets, indicating a more stable internal representation.
  • Class-Specific Gains: Some classes saw dramatic improvements. The "Bowl" class improved from 1.79% mAP to 30.87% mAP under the curriculum learning scheme, as enriched embeddings provided a much stronger signal for the visual category across different naming conventions.

Conclusion and Future Work

This research highlights that synonym relations are a critical factor in the reliability of vision-language models. While the study focused on the BARON architecture and COCO datasets, the strategies of back-translation and embedding enrichment are likely generalizable. Future work will involve testing these methods on other architectures like OWL-ViT or GLIP and exploring how synonym relations affect broader tasks like Visual Question Answering (VQA).

Summary updated

BibTeX
@inproceedings{nebbia2024synonym,
  author = {Giacomo Nebbia and Adriana Kovashka},
  title = {{Synonym relations affect object detection learned on vision-language data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.239},
}
Figure