UCPhrase: Unsupervised Context-aware Quality Phrase Tagging

Xiaotao Gu, Zihan Wang, Zhenyu Bi, Yu Meng, Liyuan Liu, Jiawei Han, Jingbo Shang

Abstract

Identifying and understanding quality phrases from context is a fundamental task in text mining. The most challenging part of this task arguably lies in uncommon, emerging, and domain-specific phrases. The infrequent nature of these phrases significantly hurts the performance of phrase mining methods that rely on sufficient phrase occurrences in the input corpus. Context-aware tagging models, though not restricted by frequency, heavily rely on domain experts for either massive sentence-level gold labels or handcrafted gazetteers. In this work, we propose UCPhrase, a novel unsupervised context-aware quality phrase tagger. Specifically, we induce high-quality phrase spans as silver labels from consistently co-occurring word sequences within each document. Compared with typical context-agnostic distant supervision based on existing knowledge bases (KBs), our silver labels root deeply in the input domain and context, thus having unique advantages in preserving contextual completeness and capturing emerging, out-of-KB phrases. Training a conventional neural tagger based on silver labels usually faces the risk of overfitting phrase surface names. Alternatively, we observe that the contextualized attention maps generated from a transformer-based neural language model effectively reveal the connections between words in a surface-agnostic way. Therefore, we pair such attention maps with the silver labels to train a lightweight span prediction model, which can be applied to new input to recognize (unseen) quality phrases regardless of their surface names or frequency. Thorough experiments on various tasks and datasets, including corpus-level phrase ranking, document-level keyphrase extraction, and sentence-level phrase tagging, demonstrate the superiority of our design over state-of-the-art pre-trained, unsupervised, and distantly supervised methods.

Introduction

In text mining, identifying "quality phrases" (such as support vector machine or heat island effect) is a foundational step for applications ranging from search engines to automated summarization. However, current methods face two significant hurdles:

  1. The Long-Tail Problem: Traditional unsupervised methods rely on corpus-wide statistics. If a phrase does not appear frequently across thousands of documents, statistical models fail to recognize it as a distinct unit. This often excludes rare or emerging technical terms.
  2. The Supervision Trap: Supervised models excel at context-aware recognition but require massive human-labeled datasets. Distant supervision using existing knowledge bases (like Wikipedia) often results in incomplete matches and cannot identify terms absent from the dictionary.

UCPhrase bridges this gap by providing an unsupervised yet context-aware framework capable of identifying phrases based on their usage within a specific document.

Methodology: The UCPhrase Framework

UCPhrase operates through a two-step process that moves away from global statistics toward document-level context and structural analysis.

The UCPhrase workflow: from silver label generation using document-level repetition to training a surface-agnostic classifier using attention maps.

1. Silver Label Generation

UCPhrase leverages the intuition that authors of technical documents repeat specific word sequences to establish terminology. The system automatically extracts "maximal" word sequences that repeat at least twice within a single document. Because these are mined per-document, they capture specific jargon even if those terms are rare in the broader corpus. These sequences serve as "silver labels"—noisy but highly relevant training data that preserves contextual completeness.

2. Surface-Agnostic Feature Extraction

Training a model directly on silver labels risks overfitting, where the model simply memorizes specific words (e.g., "New York") rather than learning the underlying structure of a phrase. To solve this, UCPhrase uses attention maps from a pre-trained Transformer (such as RoBERTa) as features.

An attention map visualization showing the structural relationships between words in a phrase, acting as a surface-agnostic fingerprint.

These maps represent how words in a sentence relate to one another. A quality phrase exhibits a distinct "structural fingerprint" in these maps regardless of the actual words used. By training a lightweight CNN on these maps, the model learns to recognize the shape of a phrase, allowing it to tag phrases it has never encountered before.

Experimental Evaluation

The researchers evaluated UCPhrase across two domains: Computer Science (KP20k dataset) and News (KPTimes dataset). The model was tested on three tasks: Phrase Ranking, Keyphrase Extraction, and Phrase Tagging.

Key Results and Insights

  • Superior Performance: On the KP20k dataset, UCPhrase achieved a phrase tagging F1 score of 73.9%, significantly outperforming the next best unsupervised method (TopMine at 40.6%) and even distantly supervised models like AutoPhrase (49.7%).
  • Context over Frequency: By focusing on document-level repetition, the model discovered 30x more unique phrases on the KP20k dataset than traditional frequency-based methods.
  • Efficiency: The study found that using only the first three layers of a Transformer provided comparable performance to using all layers, reducing computational costs by 75%.
  • Robustness to Noise: Case studies demonstrated that the model could ignore "noisy" silver labels (like typos) by assigning them low probability scores during the final tagging phase, effectively self-correcting the training data.

Conclusion

UCPhrase demonstrates that phrase mining does not have to rely on massive human labeling or high-frequency occurrences. By treating attention maps as structural X-rays of language, the model achieves a "surface-agnostic" understanding that generalizes to unseen and rare terms. Future work may explore adjusting phrase granularity (e.g., distinguishing between data mining and biomedical data mining) and applying this attention-based philosophy to other tasks like Named Entity Recognition (NER).

Summary updated

Abstract

Identifying and understanding quality phrases from context is a fundamental task in text mining. The most challenging part of this task arguably lies in uncommon, emerging, and domain-specific phrases. The infrequent nature of these phrases significantly hurts the performance of phrase mining methods that rely on sufficient phrase occurrences in the input corpus. Context-aware tagging models, though not restricted by frequency, heavily rely on domain experts for either massive sentence-level gold labels or handcrafted gazetteers. In this work, we propose UCPhrase, a novel unsupervised context-aware quality phrase tagger. Specifically, we induce high-quality phrase spans as silver labels from consistently co-occurring word sequences within each document. Compared with typical context-agnostic distant supervision based on existing knowledge bases (KBs), our silver labels root deeply in the input domain and context, thus having unique advantages in preserving contextual completeness and capturing emerging, out-of-KB phrases. Training a conventional neural tagger based on silver labels usually faces the risk of overfitting phrase surface names. Alternatively, we observe that the contextualized attention maps generated from a transformer-based neural language model effectively reveal the connections between words in a surface-agnostic way. Therefore, we pair such attention maps with the silver labels to train a lightweight span prediction model, which can be applied to new input to recognize (unseen) quality phrases regardless of their surface names or frequency. Thorough experiments on various tasks and datasets, including corpus-level phrase ranking, document-level keyphrase extraction, and sentence-level phrase tagging, demonstrate the superiority of our design over state-of-the-art pre-trained, unsupervised, and distantly supervised methods.
BibTeX
@inproceedings{gu2021ucphrase,
  author = {Xiaotao Gu and Zihan Wang and Zhenyu Bi and Yu Meng and Liyuan Liu and Jiawei Han and Jingbo Shang},
  title = {{UCPhrase: Unsupervised Context-aware Quality Phrase Tagging}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2021},
  url = {https://openreview.net/forum?id=K80_kmgDAHJ},
}
Figure