Abstract
Sentence specificity quantifies the level of detail in a sentence, characterizing the organization of information in discourse. While this information is useful for many downstream applications, specificity prediction systems predict very coarse labels (binary or ternary) and are trained on and tailored toward specific domains (e.g., news). The goal of this work is to generalize specificity prediction to domains where no labeled data is available and output more nuanced real-valued specificity ratings. We present an unsupervised domain adaptation system for sentence specificity prediction, specifically designed to output real-valued estimates from binary training labels. To calibrate the values of these predictions appropriately, we regularize the posterior distribution of the labels towards a reference distribution. We show that our framework generalizes well to three different domains with 50%-68% mean absolute error reduction than the current state-of-the-art system trained for news sentence specificity. We also demonstrate the potential of our work in improving the quality and informativeness of dialogue generation systems.
Introduction
Sentence specificity measures the degree of detail provided in a sentence—for instance, distinguishing between a general statement like "This brand is popular" and a specific one like "Mascara is the most commonly worn cosmetic, with women spending $4,000 on it." While specificity is a vital metric for summarization, argumentation mining, and dialogue systems, existing predictors face two significant hurdles. First, they exhibit high domain sensitivity; models trained on formal news corpora often fail when applied to informal text like tweets or product reviews. Second, they typically produce coarse binary or ternary outputs, failing to capture the continuous spectrum of specificity inherent in natural language. This research introduces a domain-agnostic system capable of generalizing to new domains without requiring human-labeled target data, while simultaneously producing nuanced, real-valued scores.
Methodology
The proposed framework utilizes a two-pronged architectural approach to achieve unsupervised domain adaptation and continuous output.
Self-Ensembling for Domain Adaptation
The system employs a "Student-Teacher" framework. The Student network, a BiLSTM augmented with hand-crafted shallow features, is trained on labeled source data (e.g., news). Concurrently, a Teacher network—whose weights are an exponential moving average of the Student’s—processes unlabeled target data. A consistency loss encourages the Student to match the Teacher's predictions even under input perturbations, such as Gaussian noise on word embeddings or random word deletion.
Posterior Regularization
To derive continuous values from binary training labels, the authors apply posterior regularization. Observing that specificity in natural language typically follows a Gaussian distribution, the model's output distribution is regularized to match a pre-specified target distribution derived from the source domain. This prevents the model from collapsing into binary extremes and encourages more realistic, nuanced scoring.
Datasets and Evaluation
The model was trained on a source domain of 4.3K news sentences with binary labels. For evaluation, the authors crowdsourced real-valued specificity annotations for Twitter (~1,000 tweets), Yelp (~850 reviews), and Movie Reviews (~920 sentences). Large unlabeled datasets (50K–95K) from these target domains were used for the unsupervised adaptation process.
Key Results
The proposed system demonstrated a significant performance leap, achieving a 50%–68% reduction in Mean Absolute Error (MAE) compared to the previous state-of-the-art (Speciteller) across all target domains. It also showed markedly higher Spearman and Kendall’s Tau correlations with human judgments.
Beyond accuracy, the model proved effective in improving dialogue systems. In task-based evaluations, filtering training data by specificity (removing "general" responses) outperformed filtering by sentence length, leading to more informative and diverse chatbot responses as measured by both human ratings and automated diversity metrics.
Conclusion and Future Work
This work demonstrates that real-valued, domain-agnostic specificity prediction is achievable through self-ensembling and posterior distribution matching. While the current model assumes a bell-shaped distribution of specificity—a property that held across the studied domains—future research could explore domains with different distributional characteristics. Additionally, more sophisticated, linguistically-informed data augmentation techniques could further enhance the robustness of the adaptation process.
Summary updated