TL;DR
This paper bridges the gap between corpus-level and document-level Automatic Term Extraction (ATE) by evaluating 16 state-of-the-art methods on full-length documents. They found that corpus-level performance doesn't directly translate to document-level, and while C-Value excels in technical domains, KeyConceptRelatedness emerges as a strong contender for general document-level ATE.
What Problem Does This Paper Address?
Automatic Term Extraction (ATE) is crucial for tasks like machine translation and information retrieval, often aiding translators by identifying domain-specific terminology. While many ATE methods have been developed and evaluated on large corpora, their effectiveness on individual documents – the typical scenario for translators using Computer Assisted Translation (CAT) tools – remains less understood. Existing evaluations often use simplified metrics that overlook the candidate extraction and filtering stages, making them less realistic. This paper aims to provide a more comprehensive evaluation of ATE methods specifically for document-level extraction, using a realistic setup that considers all gold terms.
Methods
The authors leverage the ATR4S toolkit, which provides a diverse set of 16 ATE methods, allowing for a direct comparison using consistent preprocessing and filtering. They categorize the ATE methods into groups based on the information they use for scoring term candidates: Frequency, Context, Reference Corpora, Topic Modeling, Wikipedia, and Re-ranking.
A key methodological contribution is their evaluation approach. Instead of relying on simplified metrics like "precision at K" that only consider a subset of gold terms, they differentiate between:
- Actual True Positives (ATP): All terms present in the gold standard.
- Recoverable True Positives (RTP): The intersection of extracted terms and gold terms.
They use two standard ATE metrics, F1 score (rank-insensitive) and average precision (AvP, rank-sensitive), calculated for both ATP and RTP levels. This provides a more nuanced understanding of method performance.
Datasets
The study utilizes three manually annotated, full-length document datasets:
- Patents: A dataset of electrical engineering patents with the fewest documents but the most terms.
- TTCm (TTC-mobile): A dataset compiled from the web under the TTC project.
- TTCw (TTC-wind): Another dataset from the TTC project, also web-crawled and filtered.
These datasets were chosen because they consist of full-length documents and are manually annotated, unlike some popular benchmark datasets that use abstracts or are not manually annotated.
Key Insights and Ideas
- Corpus vs. Document-Level Mismatch: Performance of ATE methods does not linearly transfer from corpus-level to document-level extraction. This highlights the need for specialized evaluation and potentially different methods for document-centric tasks.
- Realistic Evaluation Matters: Using ATP and RTP metrics, and considering all gold terms, provides a more accurate picture of ATE performance than simplified metrics that only look at a subset of terms.
- Domain Specificity: C-Value, a frequency-based method, performs exceptionally well on the highly technical "Patents" dataset, suggesting its strength in specialized domains.
- Hybrid Approaches Shine: KeyConceptRelatedness, which combines semantic relatedness with keyphrase extraction, emerges as a strong overall performer for document-level ATE across different domains.
- Wikipedia Features Help: Methods incorporating Wikipedia-based features generally perform better, especially when considering the more optimistic RTP metrics.
Key Results
- Corpus-Level: No single ATE method consistently outperformed others. C-Value was often a strong baseline, and Voting3 performed similarly to its more complex variant Voting5. LinkProbability generally had the lowest scores.
- Document-Level: KeyConceptRelatedness was identified as a clear overall winner. C-Value performed best on the "Patents" dataset. Statistical comparisons confirmed that C-Value significantly outperformed other methods on "Patents," while KeyConceptRelatedness significantly outperformed others on the "TTCm" and "TTCw" datasets for both F1 and AvP metrics.
- MWE Performance: C-Value modestly outperformed KeyConceptRelatedness in identifying multiword terms (MWEs) in the output.
- Document Length: Generally, as document length increased, the ATP AvP score tended to decrease or show little correlation.
Limitations and Future Work
- Limitations:
- The study acknowledges that the datasets might have a slight bias towards methods designed to extract MWEs due to the high proportion of MWEs in the gold terms.
- The "TotalTF-IDF" method was ill-equipped for document-level ATE due to its logarithmic term calculation.
- Future Work:
- Investigating ATE methods specifically tailored for single-document input, potentially by adapting Automatic Keyword and Keyphrase Extraction (AKE) methods.
- Exploring single-document bilingual ATE.
Summary updated