Abstract
This paper investigates the derivation of grounded word embeddings by leveraging the referential structure of image descriptions. Unlike traditional models that map words directly to pixels, this approach utilizes denotational context (different names for the same object) and situational context (related objects in a scene). Results demonstrate that these grounded representations capture semantic nuances and hierarchical relationships more effectively than standard text-only or purely visual models, particularly in tasks like co-reference detection and hypernymy identification.
Introduction
Modern word embeddings, such as Word2Vec or GloVe, derive meaning from textual distribution but lack physical grounding. While previous multimodal research has attempted to link words to visual features, it often overlooks the rich social and structural context inherent in how humans refer to objects. When multiple observers describe a scene, they provide a variety of labels for the same entity (e.g., "girl" vs. "young lady") and mention co-occurring objects (e.g., "cake" on a "table"). This paper exploits these referential relationships to build more precise continuous meaning representations.
Methodology
The authors propose four distinct grounding mechanisms to construct word embeddings:
- Textual Context (TXT): A baseline skip-gram model trained on the text of referring expressions.
- Visual Grounding (VIS): Words are represented by "visual prototypes"—the average visual features (extracted via GoogLeNet) of the objects they describe.
- Situational Grounding (SIT): Captures similarities between words used for different objects within the same scene, reflecting topical or environmental relationships.
- Denotational Grounding (DEN): Captures similarities between different words used for the same physical referent, identifying semantically compatible terms.
Experimental Setup
The models were trained and evaluated using the ReferIt and RefCoco datasets. Evaluation spanned several standard benchmarks, including MEN (word similarity), SemSim/VisSim (semantic/visual similarity), and BLESS (hypernymy detection). Additionally, a new Approximate Co-Reference Detection task was introduced to test if models could determine if two phrases refer to the same entity in a scene.
Key Results and Insights
The study reveals that "similarity" is multifaceted. Denotational context (DEN) excels at understanding near-interchangeability (e.g., "girl" and "lady"), whereas situational context (SIT) identifies topical associations (e.g., "grass" and "clouds").
As shown in Figure 3, the Denotational (DEN) model outperformed other grounded approaches on the MEN and SemSim datasets. When combined with textual data (DEN||TXT), it achieved high results for human similarity judgments among grounded models. Notably, the DEN model was highly effective at identifying hypernym directionality, reaching 81.14% accuracy.
Furthermore, DEN embeddings proved superior in attribute learning. Qualitative analysis showed they group related adjectives (e.g., "top," "upper") more coherently than visual models, which are often hindered by the "noise" of diverse visual appearances for abstract attributes like "small."
Conclusion
By incorporating referential structure, the authors demonstrate that grounded embeddings can capture semantic nuances often missed by purely textual or visual models. While these models currently face data-scale limitations compared to web-scale textual models, they offer a more precise framework for understanding how language relates to the physical world. Future work will focus on expanding these methods to verbs and complex spatial relations.
Summary updated