Abstract
In Natural Language Processing (NLP), the quality of word embeddings often depends on the volume of training data. However, for low-resource domains, data is scarce. This paper investigates "sentence informativeness"—the extent to which a sentence reveals the meaning of a specific word. We demonstrate that while machine-learning classifiers can effectively identify sentences that improve embedding quality, their criteria for "informativeness" differ significantly from human intuition. Using these classifiers to prioritize high-quality training sentences can significantly enhance word representations when data is limited.
Introduction
Building robust word embeddings typically requires massive corpora. For specific domains or low-resource languages, this "Big Data" requirement is a major bottleneck. This research addresses the inefficiency of treating all training sentences as equal. Some sentences are "uninformative," providing little context for a word's meaning, while others are highly descriptive. By automatically identifying and prioritizing "informative" sentences, we can potentially train high-quality models using much smaller datasets.
Methodology
The study frames informativeness as a classification task. The core hypothesis is that if a model can predict a missing word with high confidence, the surrounding sentence must be informative.
The Proxy Task
A model is presented with a sentence containing a blank slot and must distinguish the original target word from ten "distractors"—words that are linguistically similar but semantically distinct. Distractors were selected from the Corpus of Contemporary American English (COCA) and Google Unigrams to ensure similar frequencies to the target words.
Classifiers
Three primary architectures were evaluated:
- context2vec (c2v): A neural model optimized for slot-filling.
- FastText: A bag-of-ngrams classifier.
- Linguistic-based: A baseline model using 3-gram probabilities and cosine similarity.
Evaluation
Models were benchmarked against human annotations (1-5 scale of informativeness) and used to select training data for word2vec embeddings. The "gold standard" for informativeness was derived from WordNet and Cambridge Dictionary definitions.
Key Results and Insights
The experiments revealed a stark contrast between machine performance and human perception.
Classifier Performance
The c2v feed-forward neural network emerged as the most effective model for the proxy task, achieving an accuracy of 0.946. However, high accuracy on the proxy task did not translate to alignment with human judgment.
The Human-Machine Gap
There was nearly zero correlation between human informativeness ratings and classifier confidence scores. While humans prioritize semantic definitions, classifiers often favor "prototypical" sentences—those that are common or typical in structure—even if they lack deep semantic richness. For instance, the Spearman correlation for the c2v feed-forward NN was -0.162, indicating a slight inverse relationship with human intuition.
Impact on Word Embeddings
Despite the gap with human intuition, classifier-based selection proved highly effective for training embeddings. When limited to 250 sentences, those ranked "most informative" by the classifier yielded a similarity score of 0.567 against the gold standard, compared to only 0.393 for the "least informative" sentences. This suggests that "uninformative" sentences can act as noise, potentially degrading embedding quality.
Discussion
The benefits of prioritizing informative sentences are most pronounced for rare words, where every training instance is critical. For common words, the sheer volume of data makes selection less critical, but for infrequent terms, every sentence counts.
A notable limitation is the classifiers' struggle with atypical sentences. For example, dictionary definitions—highly informative to humans—may receive low scores from models trained on standard prose due to their unique linguistic structure.
Conclusion
Sentence informativeness is a powerful tool for optimizing NLP models in data-constrained environments. While machines and humans define "information" differently, machine-driven filtering effectively removes noise and prioritizes useful context. Future research will explore target-word-independent features, such as context polysemy and specific knowledge patterns, to further refine automated informativeness detection.
Summary updated