Domain Agnostic Real-Valued Specificity Prediction

Wei-Jen Ko, Greg Durrett, Junyi Jessy Li

Abstract

Sentence specificity—the level of detail provided in a text—is a critical feature for natural language processing tasks such as summarization and dialogue generation. However, existing specificity predictors often suffer from domain brittleness and coarse, binary outputs. This paper presents an unsupervised domain adaptation framework that predicts specificity as a continuous, real-valued score. By combining self-ensembling with a novel posterior regularization technique, the proposed system reduces prediction error by up to 68% compared to previous state-of-the-art models when transitioning across diverse domains like News, Twitter, and Yelp.

Introduction

In natural language processing, sentence specificity measures the granularity of information. For instance, a sentence like "This brand is popular" is considered general, whereas "Mascara is the most commonly worn cosmetic, and women will spend an average of $4,000 on it..." is highly specific.

While useful, current specificity models face two primary challenges. First, they are typically trained on formal news data and perform poorly when applied to informal domains like social media. Second, they often treat specificity as a binary classification (General vs. Specific), failing to capture the continuous spectrum of detail inherent in human language. This research addresses these gaps through unsupervised domain adaptation, leveraging binary labels from a source domain (News) and unlabeled data from target domains (Twitter, Yelp, Movie Reviews) to produce nuanced, real-valued scores.

Methodology

Base Architecture

The underlying model utilizes a Bi-directional Long Short-Term Memory (BiLSTM) network to encode sentence structure. This neural representation is concatenated with hand-crafted "shallow" features, including sentence length, punctuation density, stop word fraction, and Inverse Document Frequency (IDF) values. These combined features are then processed by a Multi-Layer Perceptron (MLP) to generate the final specificity prediction.

Domain Adaptation via Self-Ensembling

To handle the stylistic shift between domains, the authors adapt the Self-Ensembling framework. The system maintains two identical networks: a Student and a Teacher. The Student learns directly from labeled source data, while the Teacher’s weights are calculated as an exponential moving average of the Student’s weights over time.

The self-ensembling framework for unsupervised domain adaptation, utilizing labeled news data and unlabeled Twitter data.

Robustness is achieved through Consistency Loss. By applying stochastic noise to inputs (such as word deletion or Gaussian noise in embeddings), the model is trained to ensure that both the Student and Teacher produce similar predictions for the same sentence despite different noise patterns. This consistency training is applied to both labeled source data and unlabeled target data, allowing the model to learn domain-invariant features.

Posterior Regularization

A significant contribution of this work is the use of Posterior Regularization to derive real-valued outputs from binary training data. Standard binary classifiers tend to push predictions toward the extremes of 0 or 1. To counteract this, the authors introduce a regularization term that encourages the model's prediction distribution to match a "bell-shaped" reference distribution derived from human-annotated data. This effectively "stretches" the model's outputs back into a realistic continuous spectrum.

Experimental Results

Datasets and Evaluation

The model was trained on approximately 4.3K labeled news sentences (Wall Street Journal) and adapted using large unlabeled corpora from Twitter (50K), Yelp (95K), and Movie Reviews (12K). For evaluation, the authors crowdsourced real-valued [0, 1] labels for the target domains to establish a gold-standard benchmark.

Key Results and Distributional Analysis

The proposed system, denoted as SE+AD (Self-Ensembling + Adaptation/Distribution regularization), demonstrated substantial improvements over Speciteller, the previous state-of-the-art. The model achieved a 50% to 68% reduction in Mean Absolute Error (MAE) and significantly higher Spearman and Kendall’s Tau correlations with human judgments across all target domains.

Comparison of prediction distributions: The SE+AD model (black dashed) closely follows the human-annotated "real" distribution (red), whereas Speciteller (brown) clusters at the extremes.

As shown in the distributional analysis, the posterior regularization (AD) is essential for matching the human "real" distribution. Without it, models tend to produce skewed results that fail to capture the middle-ground of the specificity spectrum.

Impact on Dialogue Systems

To test practical utility, the model was integrated into a Seq2Seq chatbot. A common heuristic in dialogue systems is to filter out short responses to avoid generic replies like "I don't know." This research proves that filtering by predicted specificity is significantly more effective than filtering by length, leading to responses that human evaluators rated as more informative and higher in overall quality.

Conclusion

This work demonstrates that specificity is a continuous spectrum that can be modeled effectively across domains without target-domain labels. By combining self-ensembling with distributional priors, the SE+AD framework provides a robust tool for generating nuanced specificity scores. Future work may involve automatically estimating target distributions and applying these metrics to improve the factuality and grounding of large language models.

Summary updated

BibTeX
@inproceedings{ko2019domain,
  author = {Wei-Jen Ko and Greg Durrett and Junyi Jessy Li},
  title = {{Domain Agnostic Real-Valued Specificity Prediction}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=rm4V81Gg_aB},
}
Figure