Weakly Supervised Domain Detection

Yumo Xu, Mirella Lapata

Abstract

Identifying specific domain-heavy segments within a document is a critical yet challenging task in NLP. While document-level classification is common, pinpointing the exact sentences or phrases that justify a label—fine-grained domain detection—suffers from an annotation bottleneck. This paper introduces DETNET, a hierarchical neural framework designed to detect domain-specific sentences and words using only "weak" document-level supervision. By integrating domain definitions as prior knowledge and employing Multiple Instance Learning (MIL), DETNET achieves robust performance across languages and genres, facilitating applications such as domain-aware summarization.

Introduction

Standard text classification typically treats documents as monolithic entities. However, real-world documents often comprise a mixture of general language and domain-specific information. For instance, a legal news report may blend common narrative with specialized terminology. Labeling every sentence in large-scale datasets is prohibitively expensive. To address this, we propose the task of fine-grained domain detection using weak supervision, where the model learns to identify "needles" of domain evidence in a "haystack" of general text.

Methodology: The DETNET Framework

The proposed model, DETNET (Domain Detection Network), utilizes an encoder-detector architecture designed to capture context at multiple granularities.

Encoder-Detector Architecture

DETNET consists of a hierarchical encoder and a corresponding detector. The encoder, based on the Transformer architecture, processes text at the word, sentence, and document levels. The detector then uses these representations to assign domain scores.

Overview of the DETNET architecture, showing the interaction between the hierarchical encoder, domain definitions, and the multi-level detector.

Multiple Instance Learning and Scoring

The core of DETNET's weak supervision is Multiple Instance Learning (MIL). The model treats the document as a "bag" and its constituent sentences and words as "instances." It learns to attribute the document-level label to specific instances by calculating hierarchical scores.

The hierarchical scoring mechanism of DETNET, illustrating how word and sentence scores are computed and aggregated to form a document-level prediction.

A key innovation is the integration of prior knowledge. The Word Detector incorporates domain definitions (short descriptive texts) to guide the identification of domain-heavy words, even in unfamiliar contexts. A gated mechanism balances signals from the raw input text against these external definitions, enhancing the model's generalization capabilities.

Experimental Setup and Datasets

The model was primarily trained on Wikipedia datasets in both English and Chinese, covering seven domains: Business, Government, Health, Law, Lifestyle, Military, and General Purpose. To evaluate sentence-level precision without gold-standard labels, the authors created a "Silver Standard" synthetic dataset by mixing domain-heavy lead sentences. Generalization was further tested on the New York Times (NYT) corpus, and human evaluations were conducted via Amazon Mechanical Turk to validate word and sentence-level detections.

Key Results

  • Sentence-Level Performance: DETNET achieved a 54.37% Macro-F1 score on English Wikipedia sentences, significantly outperforming standard MIL baselines (37.12%) and hierarchical models without word-level signals (30.01%).
  • Cross-Genre Generalization: The model maintained a strong 60.62% F1 score on the NYT dataset, demonstrating that the inclusion of domain definitions prevents overfitting to training-specific vocabulary.
  • Qualitative Word Retrieval: DETNET successfully identified highly specific domain terms (e.g., "biosecurity") compared to broader topic modeling approaches like Labeled LDA.
  • Application in Summarization: The authors introduced DETRANK, a domain-aware summarization tool. Human evaluators preferred DETRANK's summaries for their informativeness and coherence compared to standard graph-based methods.

Conclusion

DETNET demonstrates that hierarchical modeling combined with weak supervision and prior knowledge is highly effective for fine-grained domain detection. Future work may explore semi-supervised learning to further refine performance and address label ambiguity in multi-domain contexts.

Summary updated

Abstract

In this paper we introduce domain detection as a new natural language processing task. We argue that the ability to detect textual segments which are domain-heavy, i.e., sentences or phrases which are representative of and provide evidence for a given domain could enhance the robustness and portability of various text classification applications. We propose an encoder-detector framework for domain detection and bootstrap classifiers with multiple instance learning (MIL). The model is hierarchically organized and suited to multilabel classification. We demonstrate that despite learning with minimal supervision, our model can be applied to text spans of different granularities, languages, and genres. We also showcase the potential of domain detection for text summarization.
BibTeX
@article{xu2019weakly,
  author = {Yumo Xu and Mirella Lapata},
  title = {{Weakly Supervised Domain Detection}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2019},
  doi = {10.1162/tacl\_a\_00287},
}
Figure