Describe Me an Auklet: Generating Grounded Perceptual Category Descriptions

Bill Noble, Nikolai IlinykhEMNLP 2023 · DOI

Introduction: Beyond Instance-Level Captioning

Most Vision-and-Language (V&L) models are currently evaluated on instance-level tasks, such as generating a caption for a specific image. However, human communication is far more flexible. We often describe abstract categories to help others identify objects they have never seen before—a process psychologists call "fast mapping." For example, if someone asks "What is a Crested Auklet?", a helpful response describes general features like a "bright orange beak" and "musk gray body" rather than describing a single specific bird on a specific rock.

Conceptual overview of the Describer-Interpreter task: transferring visual category knowledge through natural language.

This paper investigates whether neural networks can achieve "communicative success" by passing visual knowledge through the bottleneck of natural language. The goal is to determine if a model can generate a "field guide" description discriminative enough for another model to recognize a novel category.

The Describer-Interpreter Framework

To test this, the authors propose a framework consisting of two distinct agents:

  1. The Describer (GEN): This model generates a text description ($u$) of a visual category. The researchers explored different ways for GEN to represent a category: using an abstract "average" vector (Prototypes) or features from a specific representative image (Exemplars).
  2. The Interpreter (IPT): This model acts as the student. It receives the generated text $u$ and attempts zero-shot classification—identifying images ($x$) of the described category without having seen any images of that category during its training.
The GEN (Describer) and IPT (Interpreter) architectures used to evaluate communicative success.

The framework evaluates the "communicative success" of the Describer based on the Interpreter's accuracy in identifying the correct category among unseen classes.

Experimental Setup

The study utilized the CUB-200-2011 (Caltech-UCSD Birds) dataset, which contains 11,788 images across 200 bird species. The data was split into 180 "seen" classes for training and 20 "unseen" classes for testing. Human-written descriptions (10 per image) provided the ground truth for supervision, allowing the models to learn the relationship between visual features and descriptive language.

Key Findings and Insights

Exemplars vs. Prototypes

Models using specific "exemplar" images to generate descriptions significantly outperformed those using abstract "prototypes." This suggests that specific visual details are more communicative and easier for the Interpreter to leverage than smoothed-out category averages.

The "Safe" Description Trap

A major finding was the impact of decoding strategies. Beam Search, while producing grammatically correct and high-probability text, often resulted in "safe" but generic descriptions (e.g., "this is a bird with wings"). These descriptions achieved high scores on standard NLP metrics like BLEU but were useless for classification because they lacked discriminative features. Conversely, Nucleus Sampling produced more descriptive text, but the Interpreter often struggled to parse its less standard phrasing.

The Metric Mismatch

The researchers identified a significant "metric mismatch." Traditional text-generation metrics like CIDEr and BLEU-4 did not correlate well with the Interpreter's accuracy. A model could produce text that looks like a human description to an automated metric but fails to convey the necessary information for recognition. This highlights the necessity of task-based evaluation in natural language generation.

Conclusion

This research demonstrates that while current models can generate descriptions of novel categories, there is a significant gap between linguistic fluency and communicative utility. True grounding requires the ability to use language to transfer visual concepts effectively. While the best model (GEN-EX) achieved 8.6% accuracy—well above the 0.5% random baseline—it still falls short of the 19.1% achieved using human descriptions, suggesting ample room for improvement in how models generalize visual concepts into language.

Summary updated

BibTeX
@inproceedings{noble2023describe,
  author = {Bill Noble and Nikolai Ilinykh},
  title = {{Describe Me an Auklet: Generating Grounded Perceptual Category Descriptions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.580},
}
Figure