Evaluating Automatic Term Extraction Methods on Individual Documents

Antonio Sajatovic, Maja Buljan, Jan Snajder, Bojana Dalbelo BasicMWE-WN @ ACL 2019 · DOI

TL;DR

This paper bridges the gap between corpus-level and document-level Automatic Term Extraction (ATE) by evaluating 16 state-of-the-art methods on full-length documents. They found that corpus-level performance doesn't directly translate to document-level, and while C-Value excels in technical domains, KeyConceptRelatedness emerges as a strong contender for general document-level ATE.

What Problem Does This Paper Address?

Automatic Term Extraction (ATE) is crucial for tasks like machine translation and information retrieval, often aiding translators by identifying domain-specific terminology. While many ATE methods have been developed and evaluated on large corpora, their effectiveness on individual documents – the typical scenario for translators using Computer Assisted Translation (CAT) tools – remains less understood. Existing evaluations often use simplified metrics that overlook the candidate extraction and filtering stages, making them less realistic. This paper aims to provide a more comprehensive evaluation of ATE methods specifically for document-level extraction, using a realistic setup that considers all gold terms.

Methods

The authors leverage the ATR4S toolkit, which provides a diverse set of 16 ATE methods, allowing for a direct comparison using consistent preprocessing and filtering. They categorize the ATE methods into groups based on the information they use for scoring term candidates: Frequency, Context, Reference Corpora, Topic Modeling, Wikipedia, and Re-ranking.

A key methodological contribution is their evaluation approach. Instead of relying on simplified metrics like "precision at K" that only consider a subset of gold terms, they differentiate between:

  • Actual True Positives (ATP): All terms present in the gold standard.
  • Recoverable True Positives (RTP): The intersection of extracted terms and gold terms.

They use two standard ATE metrics, F1 score (rank-insensitive) and average precision (AvP, rank-sensitive), calculated for both ATP and RTP levels. This provides a more nuanced understanding of method performance.

Datasets

The study utilizes three manually annotated, full-length document datasets:

  • Patents: A dataset of electrical engineering patents with the fewest documents but the most terms.
  • TTCm (TTC-mobile): A dataset compiled from the web under the TTC project.
  • TTCw (TTC-wind): Another dataset from the TTC project, also web-crawled and filtered.

These datasets were chosen because they consist of full-length documents and are manually annotated, unlike some popular benchmark datasets that use abstracts or are not manually annotated.

Key Insights and Ideas

  • Corpus vs. Document-Level Mismatch: Performance of ATE methods does not linearly transfer from corpus-level to document-level extraction. This highlights the need for specialized evaluation and potentially different methods for document-centric tasks.
  • Realistic Evaluation Matters: Using ATP and RTP metrics, and considering all gold terms, provides a more accurate picture of ATE performance than simplified metrics that only look at a subset of terms.
  • Domain Specificity: C-Value, a frequency-based method, performs exceptionally well on the highly technical "Patents" dataset, suggesting its strength in specialized domains.
  • Hybrid Approaches Shine: KeyConceptRelatedness, which combines semantic relatedness with keyphrase extraction, emerges as a strong overall performer for document-level ATE across different domains.
  • Wikipedia Features Help: Methods incorporating Wikipedia-based features generally perform better, especially when considering the more optimistic RTP metrics.

Key Results

  • Corpus-Level: No single ATE method consistently outperformed others. C-Value was often a strong baseline, and Voting3 performed similarly to its more complex variant Voting5. LinkProbability generally had the lowest scores.
  • Document-Level: KeyConceptRelatedness was identified as a clear overall winner. C-Value performed best on the "Patents" dataset. Statistical comparisons confirmed that C-Value significantly outperformed other methods on "Patents," while KeyConceptRelatedness significantly outperformed others on the "TTCm" and "TTCw" datasets for both F1 and AvP metrics.
  • MWE Performance: C-Value modestly outperformed KeyConceptRelatedness in identifying multiword terms (MWEs) in the output.
  • Document Length: Generally, as document length increased, the ATP AvP score tended to decrease or show little correlation.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that the datasets might have a slight bias towards methods designed to extract MWEs due to the high proportion of MWEs in the gold terms.
  • The "TotalTF-IDF" method was ill-equipped for document-level ATE due to its logarithmic term calculation.
  • Future Work:
    • Investigating ATE methods specifically tailored for single-document input, potentially by adapting Automatic Keyword and Keyphrase Extraction (AKE) methods.
  • Exploring single-document bilingual ATE.

Summary updated

BibTeX
@inproceedings{sajatovic2019evaluating,
  author = {Antonio Sajatovic and Maja Buljan and Jan Snajder and Bojana Dalbelo Basic},
  title = {{Evaluating Automatic Term Extraction Methods on Individual Documents}},
  booktitle = {MWE-WN @ ACL},
  year = {2019},
  doi = {10.18653/v1/w19-5118},
}
Figure