TL;DR
This paper explores whether neural image classifiers can mimic human visual metaphor comprehension by treating misclassifications as potential metaphors. They propose that by analyzing the visual features encoded by these models, we can gain insights into how visually grounded metaphors are constructed and processed.
What Problem Does This Paper Address?
Understanding how humans create and comprehend metaphors, especially those grounded in visual similarity, remains a challenge. While neurolinguistic studies suggest sensory knowledge plays a role, computational models have largely focused on linguistic cues. This paper bridges this gap by investigating if modern image classifiers, trained on vast visual datasets, exhibit behaviors analogous to human visual metaphor generation, particularly when encountering novel or ambiguous visual information. The core question is whether the "misinterpretations" of these classifiers can be seen as a form of computational metaphor.
Methods
The researchers employ pre-trained convolutional neural networks (CNNs) like ResNet50 and VGG16, commonly used for image classification and captioning. Their approach involves two main experiments:
- Mis-categorization Analysis: They feed images into these classifiers and analyze the resulting probability distributions for different object categories. The hypothesis is that the top mis-classifications might reveal visual similarities that humans would use metaphorically.
- Visual Space Exploration: They leverage the intermediate feature representations (vectors) generated by the CNNs before the final classification layer. By treating these vectors as points in a multi-dimensional "visual space," they explore how combining or comparing these vectors can model metaphorical compositionality. This involves calculating cosine similarities between these visual vectors to quantify relationships.
Datasets
The paper uses a custom dataset of approximately 100 images that human users have described using metaphors or similes. Crucially, these metaphors were selected such that the target of the metaphor was present in the ImageNet ontology (the categories the models were trained on), while the source might not be. This allows for the analysis of mis-classifications where the model fails to identify the intended object but might offer a visually related alternative.
Key Insights and Ideas
- Misclassifications as Potential Metaphors: Image classifiers, when faced with unfamiliar objects or unusual presentations, often produce mis-classifications that can surprisingly align with human metaphorical descriptions. For instance, a lightning strike being classified as a "spider web" due to their branching, thread-like appearance.
- Visual Feature Transfer: The models' mis-classifications suggest they are transferring visual features from known categories to unknown ones, mirroring how metaphors transfer properties from a source to a target.
- Visual Space for Compositionality: The intermediate feature vectors from CNNs form a "visual space" where concepts can be represented and manipulated. Combining vectors (e.g., "sky" + "fire") can create new vectors that are closer to the target metaphor ("sky on fire") than the original source vector, demonstrating a form of computational metaphor composition.
- Limitations of Single-Word Captions: Standard image captioning models often produce single-word outputs, which limits their ability to capture the compositional nature of many human metaphors (e.g., "lawn is a green carpet").
Key Results
- Metaphorical Alignment: In a dataset of human-described metaphors, the top mis-classifications from CNNs showed a notable overlap (around 30% for the top 5 captions) with human metaphorical descriptions, suggesting a shared basis in visual similarity.
- Compositionality Demonstrated: By summing visual vectors for "sky" and "fire," the resulting vector showed significantly higher cosine similarity to the "sky on fire" sunset vector than the "sky" vector alone, indicating that adding the "fire" modifier computationally recreated the metaphor.
- F-Scores for Compositionality: In evaluating metaphorical compositionality, models achieved F-scores around 0.6-0.7, indicating that adding a modifier vector often improved the similarity between source and target vectors for real metaphors, while often worsening it for false metaphors.
Limitations and Future Work
- Small Datasets: The authors acknowledge that their datasets for both mis-classification analysis and compositional metaphors are small, limiting the generalizability of their findings.
- Limited Ontology: The models' pre-defined ontologies (e.g., ImageNet categories) restrict the types of metaphors that can be analyzed, particularly those involving sources absent from the training data.
- Focus on Visual Similarity: The current approach primarily captures visual similarities. Future work could explore how extra-visual knowledge or more complex linguistic structures contribute to metaphor comprehension.
- Expanding Datasets: A key future direction is collecting larger, more diverse corpora of visually grounded metaphors and their corresponding images.
- Investigating Visual Clues: Further research could identify which specific visual features are most influential for classifiers when reproducing human metaphorical combinations.
- Human Appreciation: Exploring how humans perceive and appreciate computationally generated metaphors is also a promising avenue.
Summary updated