Abstract
This research explores the concept of "sentence informativeness"—the degree to which a sentence provides sufficient context to help infer a word's meaning. While modern NLP relies on massive datasets, specific domains often suffer from data scarcity. The authors demonstrate that identifying and prioritizing informative sentences can significantly improve word embedding quality in low-resource scenarios, despite a notable discrepancy between human intuition and machine-predicted informativeness.
Introduction: The Value of Context
Modern Natural Language Processing (NLP) depends heavily on word embeddings, which are mathematical representations of word meanings. Typically, these are trained on vast corpora like Wikipedia. However, in specialized fields such as medicine or law, or for low-resource languages, data is often limited. In these "data-hungry" scenarios, the quality of every sentence matters.
Some sentences provide zero context (e.g., "I went to the ***"), while others are rich with semantic clues (e.g., "I went to the *** to withdraw money"). The goal of this research is to automatically identify these high-value sentences to build more robust models from smaller datasets.
Methodology
The researchers approached informativeness through two primary lenses: machine learning classification and human judgment.
1. The Classification Task
Informativeness was framed as a "fill-in-the-blank" problem. If a model can accurately guess a masked word based on its context, the sentence is considered informative. To ensure the task was rigorous, the authors introduced "distractors"—words that are statistically likely in similar contexts but are not synonyms (e.g., using "supermarket" as a distractor for "bank").
2. Algorithms and Human Labeling
Several classifiers were tested, ranging from linguistic n-gram models to neural approaches like context2vec (c2v), which uses bidirectional LSTMs to represent the target word's slot. Simultaneously, human annotators rated sentences on a scale of 1 to 5 based on how much information they provided about a masked word’s meaning.
The Human-Machine Discrepancy
A striking finding of the study was the near-zero correlation between human intuition and machine predictions. A classifier might assign a high informativeness score to a statistically common but semantically vague sentence like "What can I tell him?" because the word "tell" is highly probable in that structure. Conversely, humans found dictionary definitions highly informative, while classifiers struggled with them due to their unique grammatical structures compared to natural usage.
Despite this gap, the machine-predicted scores proved to be effective quality filters. The models identified sentences where the target word was statistically "at home," effectively isolating signal from noise.
Results and Validation
Classifier Performance
The context2vec feed-forward neural network emerged as the top performer, achieving 94.6% accuracy in the classification task. As shown in the performance summary, this model significantly outperformed baselines, though it maintained a low correlation with human judgments.
Impact on Word Embeddings
To validate the utility of these scores, the authors used the classifier to select training data for a word2vec model. Embeddings trained on the top 250 "informative" sentences achieved a similarity score of 0.567, compared to just 0.393 for the 250 least informative sentences. Crucially, adding uninformative sentences to the training set actually decreased embedding quality, suggesting that in low-data regimes, data quality is more important than quantity.
Conclusion and Future Work
The study highlights that "informativeness" is a subjective and complex metric, with even human annotators showing relatively low agreement (Spearman correlation of ~0.33). However, the success of machine-driven informativeness scoring opens new avenues for efficient NLP in specialized domains. Future research aims to develop "universal" informativeness scorers that are independent of specific target words, potentially by analyzing context word polysemy or topical density.
Summary updated