Knowledge Supports Visual Language Grounding: A Case Study on Colour Terms

Simeon Schüz, Sina ZarrießACL 2020 · DOI

Abstract

Grounding color terms in real-world images is a complex task for artificial intelligence due to environmental factors like lighting and shadows. This paper investigates the role of "world knowledge"—human-like priors about typical object colors—in improving computational color grounding. We evaluate four modeling strategies, ranging from purely visual bottom-up approaches to integrated early and late fusion models. Our results show that incorporating semantic knowledge significantly enhances performance on color-diagnostic objects, though challenges remain in accurately identifying atypical color instances.

Introduction

In computer vision, "grounding" is the process of linking linguistic terms, such as "green," to specific pixel regions in an image. Traditional models often rely on a bottom-up approach, deriving labels solely from visual features like color histograms. However, human perception is augmented by the Memory Colour Effect, where prior knowledge of an object's typical color (e.g., bananas are yellow) helps maintain color constancy under difficult lighting conditions.

Visual grounding of the term 'green tree' in a complex scene.

This research aims to bridge the gap between purely visual systems and human-like perception by integrating object-specific knowledge into the color-naming process.

Methodology

The study compares four distinct modeling strategies to evaluate the impact of world knowledge:

  1. Bottom-Up: A baseline feed-forward network predicting color terms from 3D RGB histograms of object crops.
  2. Top-Down: A model predicting color based solely on the object's name (using GloVe word embeddings), representing the "prior" or "canonical" color.
  3. Late Fusion: A rigid integration where the outputs of the Bottom-Up and Top-Down models are averaged.
  4. Early Fusion: A learned integration where object embeddings and visual features are concatenated early in the network, allowing for complex interactions between visual evidence and prior knowledge.
Comparison of Late Fusion and Early Fusion architectures.

Experimental Setup

The researchers utilized the VisualGenome dataset, which contains over 100,000 images and 3.8 million annotated objects. To analyze the influence of knowledge, objects were categorized by color entropy:

  • Color Diagnostic Objects (CDOs): Predictable colors (e.g., carrots, trees).
  • Color Neutral Objects (CNOs): Variable colors (e.g., umbrellas, balloons).
  • Color Biased Objects (CBOs): Intermediate predictability (e.g., stone).

Key Insights and Results

The integration of world knowledge provided several critical advantages:

  • Knowledge as a Powerful Prior: The Top-Down model alone achieved 72.6% accuracy on CDOs, significantly outperforming the pixel-only Bottom-Up model (54%).
  • The Challenge of Atypicality: A major hurdle is identifying "atypical" cases, such as a red tree. Late Fusion often fails here because the rigid prior (trees are green) overrides visual evidence.
  • Superiority of Early Fusion: Early Fusion proved more flexible, achieving 15.7% accuracy on atypical CDOs—more than double the 6.9% achieved by Late Fusion. This suggests that learned integration better balances visual evidence with conceptual priors.
  • Zero-Shot Generalization: By leveraging semantic word embeddings, the models demonstrated an ability to generalize to unseen object types (e.g., inferring properties of "limes" based on knowledge of "lemons"), achieving 32-38% accuracy on novel categories.

Conclusion and Future Work

While integrating world knowledge boosts overall accuracy from 45.0% to 52.1%, models still exhibit a strong "canonical bias," occasionally favoring typical colors over clear visual evidence. Future research will focus on more sophisticated fusion strategies and embedding color prediction more deeply into broader object recognition and scene understanding pipelines.

Summary updated

BibTeX
@inproceedings{schuz2020knowledge,
  author = {Simeon Schüz and Sina Zarrieß},
  title = {{Knowledge Supports Visual Language Grounding: A Case Study on Colour Terms}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.584},
}
Figure