Domain Agnostic Real-Valued Specificity Prediction

Wei-Jen Ko, Greg Durrett, Junyi Jessy Li

TL;DR

This paper introduces an unsupervised domain adaptation framework that predicts continuous, real-valued sentence specificity across diverse domains (like Twitter and Yelp) using only binary-labeled news data for training. By combining self-ensembling with posterior distribution regularization, the system reduces prediction errors by up to 68% compared to existing baselines and demonstrates practical utility by significantly improving open-domain dialogue generation.

What Problem Does This Paper Address?

Sentence specificity—the level of detail contained within a sentence—is a foundational element of discourse analysis. It is highly valuable for downstream NLP tasks like text summarization, deception detection, and dialogue generation.

However, existing specificity prediction systems suffer from two major bottlenecks:

  1. Coarse Categorization: They typically treat specificity as a binary ("general" vs. "specific") or ternary classification problem, whereas human language naturally exhibits a continuous spectrum of detail.
  2. Domain Sensitivity: Existing models are heavily tailored to the specific domains they were trained on (usually news articles) and perform poorly when deployed on vastly different text styles, such as social media or user reviews.

Acquiring continuous, human-annotated specificity labels for every new domain is expensive and impractical. This paper addresses the challenge of training a model on binary data from a source domain (news) and adapting it to predict nuanced, real-valued specificity in target domains without any labeled target data.

Methods

The authors propose an unsupervised domain adaptation framework that merges two key components:

1. Self-Ensembling for Text

Derived from computer vision techniques, this architecture uses a Student-Teacher network paradigm to learn robust, domain-invariant features:

  • Base Model: Sentences are encoded using a bidirectional LSTM combined with hand-crafted shallow features (e.g., token count, fraction of stop words, sentiment polarity, and IDF scores), which are then processed by a Multi-Layer Perceptron (MLP).
  • Temporal Ensembling: The teacher network's parameters are updated as an exponential moving average of the student network's parameters.
  • Text Noise Augmentation: To force the networks to learn stable representations across domains, distinct noise is applied to the inputs of both networks. Since text is highly sensitive to perturbation, the authors designed specific text augmentations: adding Gaussian noise to word embeddings, randomly removing words, and substituting word vectors. A consistency loss (squared difference) ensures both networks yield similar predictions despite the noise.

2. Posterior Regularization

Because the training data contains only binary labels, standard classifiers tend to output extreme probabilities (pushing predictions close to 0 or 1). Furthermore, consistency losses can cause predictions to collapse toward a narrow mean value.

To counteract this, the authors introduce a posterior regularization term. This forces the model's predicted distribution to match a pre-specified, bell-shaped reference distribution derived from the source domain. They experiment with two formulations—one using KL divergence assuming Gaussian distributions, and a simpler one matching only the mean and standard deviation—and find both to be highly effective at shaping realistic, real-valued outputs.

Datasets

  • Source Domain (News): A combined corpus of 4.3K sentences with binary labels (general vs. specific) compiled from the Wall Street Journal and other existing news datasets.
  • Target Domains (Evaluation): The authors crowdsourced continuous specificity annotations (scaled from 0 to 1) for three target domains to evaluate generalization:
  • Twitter: 984 annotated tweets (plus 50K unlabeled tweets for adaptation).
  • Yelp Reviews: 845 annotated sentences (plus 95K unlabeled).
  • Movie Reviews: 920 annotated sentences (plus 12K unlabeled).

Key Insights and Ideas

  • The Balancing Act of Regularization: Unsupervised domain adaptation via self-ensembling naturally pulls predictions together. Without posterior regularization, the model's outputs tightly cluster around a narrow, uninformative range (between 0.4 and 0.6). The regularization term acts as a vital counter-force, stretching the predictions out to match real-world distributions.
  • Cross-Domain Distribution Consistency: The authors discovered that the mean and variance of sentence specificity are remarkably consistent across news, tweets, and reviews. This stability justifies using the source domain's distribution as a universal reference anchor.
  • Specificity as a Better Filter than Length: In open-domain dialogue systems, a common trick to avoid generic responses (like "I don't know") is to filter out short training sentences. The authors show that filtering out low-specificity sentences is a fundamentally superior proxy for informativeness, leading to much higher response diversity.

Key Results

  • Error Reduction: The full system (combining self-ensembling and posterior regularization) achieved a 50% to 68% reduction in Mean Absolute Error (MAE) compared to Speciteller, the state-of-the-art news-trained baseline, across all target domains.
  • Human Correlation: The model significantly outperformed baselines in Spearman and Kendall’s Tau correlations, proving its real-valued predictions align closely with human perception.
  • Dialogue Generation Gains: When training a Seq2Seq chatbot on the OpenSubtitles dataset, filtering out generic responses using the proposed model resulted in lower perplexity, higher unigram/bigram diversity, and significantly higher human evaluation scores for both quality and informativeness compared to length-based filtering.

Limitations and Future Work

  • Sensitivity of Text Augmentation: Text data is inherently fragile under noise augmentation compared to images. The authors note that finding the exact right balance and types of text distortion remains a challenge.
  • Assumption of Distribution Alignment: The framework relies on the target domain's true specificity distribution mirroring the source domain's bell curve. If a target domain naturally possesses a highly skewed or bimodal specificity distribution, this regularization technique might artificially distort the predictions.

Summary updated

Abstract

Sentence specificity quantifies the level of detail in a sentence, characterizing the organization of information in discourse. While this information is useful for many downstream applications, specificity prediction systems predict very coarse labels (binary or ternary) and are trained on and tailored toward specific domains (e.g., news). The goal of this work is to generalize specificity prediction to domains where no labeled data is available and output more nuanced real-valued specificity ratings. We present an unsupervised domain adaptation system for sentence specificity prediction, specifically designed to output real-valued estimates from binary training labels. To calibrate the values of these predictions appropriately, we regularize the posterior distribution of the labels towards a reference distribution. We show that our framework generalizes well to three different domains with 50%~68% mean absolute error reduction than the current state-of-the-art system trained for news sentence specificity. We also demonstrate the potential of our work in improving the quality and informativeness of dialogue generation systems.
BibTeX
@inproceedings{ko2019domain,
  author = {Wei-Jen Ko and Greg Durrett and Junyi Jessy Li},
  title = {{Domain Agnostic Real-Valued Specificity Prediction}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=rm4V81Gg_aB},
}

Loading related content...

Figure