Abstract
Grounded language learning faces a significant challenge known as the "no negative evidence" problem: human descriptions of objects typically focus on what an object is rather than what it is not. This paper proposes an unsupervised framework to automatically identify semantically distant negative examples using TF-IDF for term selection and Paragraph Vectors for similarity mapping. By leveraging linguistic descriptions as a proxy for visual distance, the system bootstraps high-quality training data for visual classifiers, consistently outperforming traditional random sampling methods.
Introduction
Grounded language learning aims to bridge the gap between linguistic constructs and perceptual data, such as pixels or depth information. A major hurdle in this field is that natural language datasets are overwhelmingly "positive-only." When asked to describe a banana, a person will likely say it is "yellow" and "curved," but they almost never specify that it is "not blue" or "not a cube."
For a machine learning model to learn the boundaries of a concept like "yellow," it requires negative examples to prevent over-generalization. While manual labeling is expensive and random sampling is noisy, this paper introduces a method to automatically "bootstrap" negative examples from unlabelled perceptual and linguistic data.
Unsupervised Negative Selection
The authors propose a two-step unsupervised pipeline to identify high-quality negative training data:
- Term Selection (TF-IDF): The system uses Term Frequency-Inverse Document Frequency (TF-IDF) to identify words that are highly descriptive of specific objects while filtering out visually uninformative "stop words."
- Semantic Mapping (Paragraph Vectors): Once a target term (e.g., "carrot") is selected, the system uses Paragraph Vectors (PV-DM) to map variable-length descriptions into fixed-length feature vectors.
- Similarity Mapping: By calculating the cosine similarity between these vectors, the system identifies objects with the most semantically dissimilar descriptions. These "distant" objects are then used as negative training data for visual classifiers.
Dataset and Visual Grounding
The researchers introduced a new dataset comprising 72 unique objects across 18 classes, including food items and geometric blocks. RGB-D images were captured from multiple angles, paired with 3,055 natural language descriptions collected via Amazon Mechanical Turk. Visual classifiers were trained using logistic regression on features extracted from the RGB-D data, including color histograms and 3D shape descriptors.
Key Results and Insights
The core insight of this work is that language serves as an effective proxy for visual distance. If two objects are described in very different ways, they are likely visually distinct in ways useful for training.
- Human Alignment: The automated similarity metric (Paragraph Vectors) agreed with human judgments of object similarity in 84% of cases.
- Classifier Performance: Classifiers trained using these automatically selected negatives consistently outperformed baselines using random negative sampling or all non-overlapping objects.
- Domain Strengths: Color classifiers performed exceptionally well (e.g., "blue" achieved 0.995 accuracy), while shape classifiers (e.g., "arch" at 0.532) proved more challenging due to linguistic inconsistency among annotators and viewpoint variance.
Conclusion and Future Work
This research demonstrates that semantic bootstrapping can effectively address the lack of negative evidence in grounded language learning. However, limitations remain, particularly regarding the complexity of shape descriptions and the current "word-as-classifier" approach which ignores syntax and negation. Future work may involve integrating this system into conversational agents that can proactively seek negative examples to resolve semantic ambiguity.
Summary updated