TL;DR
This paper introduces a novel method for automatically inducing "synsets" (groups of words with the same meaning) by combining visual and textual information from images and their associated text. Their multi-modal approach significantly outperforms uni-modal methods and achieves quality comparable to human-annotated resources like ImageNet, without requiring manual annotation.
What Problem Does This Paper Address?
Understanding natural language is challenging due to polysemy (words with multiple meanings) and synonymy (different words with the same meaning). While resources like WordNet manually organize word meanings into synsets, creating and maintaining them is labor-intensive and limited in scope. Existing methods for "word sense induction" (identifying different meanings of a word) primarily rely on textual context. This paper tackles the problem of automatically creating a comprehensive, multi-modal synset resource by jointly leveraging both visual and textual information, going beyond existing work that either focuses solely on text or assumes word senses are already known.
Methods
The core of the paper is a two-stage process:
- Polysemy Detection (Word Sense Induction): For each noun phrase, the system clusters its associated multi-modal observations (visual features from VGG and textual features from LSA) to identify distinct senses. This is achieved using k-means clustering, with the optimal number of clusters determined by the gap statistic. A constraint is enforced to ensure each induced sense has a minimum number of observations.
- Synonymy Detection (Synset Induction): After inducing word senses, the system groups similar senses into synsets. Instead of using the gap statistic (which is less suitable for a known number of synsets), they employ a greedy merging approach. This involves computing mean vectors for each induced sense and iteratively merging the closest ones until a desired number of synsets is reached.
The novelty lies in the multi-modal fusion of visual and textual features. They use an "early fusion" paradigm, combining features with a weighted cosine distance, allowing them to balance the influence of each modality.
Datasets
- ImageNet: The authors use a subset of ImageNet synsets (specifically, those not used to train the VGG network) as their primary source. They extract images and associated noun phrases.
- Web Text: For each ImageNet image, they perform a reverse image search to retrieve associated web text, which is then processed using Latent Semantic Analysis (LSA) to create textual embeddings.
- New Benchmark: The paper introduces a new dataset derived from ImageNet, consisting of image-text pairs labeled with noun phrases, along with pre-computed VGG visual features and LSA textual features. This dataset is made publicly available.
Key Insights and Ideas
- Multi-modal is better: Combining visual and textual information for synset induction significantly outperforms using either modality alone. This suggests that visual context provides complementary information that helps disambiguate word senses and group similar meanings.
- Unsupervised quality: The unsupervised, multi-modal approach can produce synsets that are remarkably close in quality to human-annotated resources like ImageNet, even without explicit manual annotation.
- ImageNet's noise: Human evaluations reveal that ImageNet synsets, despite being human-created, are not always perfectly sensible and can sometimes be noisy or overly granular. The unsupervised method sometimes groups concepts that humans find more sensible than ImageNet's distinctions.
- Balancing act: The multi-modal approach effectively balances the tendencies of uni-modal methods: vision-only methods can overproduce synsets, while text-only methods can under-produce them.
Key Results
- Automated Evaluation: The multi-modal approach achieved the highest v-measure and was tied for the highest paired f-measure in automated evaluations compared to uni-modal approaches.
- Human Evaluation: Human annotators rated the multi-modal induced synsets as sensible approximately 84% as often as ImageNet synsets. This indicates a high level of understandability for the unsupervised synsets.
- Comparison to Uni-modal: Human judges did not significantly favor ImageNet synsets over multi-modal induced synsets, but they did favor ImageNet's over uni-modally induced synsets.
Limitations and Future Work
- Limitations:
- The dataset construction involved an arbitrary division of observations among noun phrases, which might not reflect realistic sense distributions.
- The current method relies on pre-trained VGG features and LSA, which might not be optimal for all types of data.
- The paper notes that ImageNet synsets are not always perfectly sensible, and their own induced synsets might also have such issues.
- Future Work:
- Exploring alternative clustering methods, such as non-parametric hierarchical agglomerative clustering, for synonymy detection.
- Investigating different weighting schemes for combining visual and textual modalities.
- Applying the multi-modal synset induction to other domains and languages, and to less categorical labels than noun phrases (e.g., abstract action concepts).
- Extending the approach to grounded language systems where word meanings are associated with real-world object properties.
Summary updated