TL;DR
This paper presents an improved automatic keyword extraction system by refining existing methods and introducing a regression-based approach. The enhancements lead to better keyword identification, particularly by treating keywordness as a continuous value rather than a binary classification, ultimately achieving more consistent results compared to human indexers.
What Problem Does This Paper Address?
Automatic keyword extraction aims to identify a concise set of terms that accurately represent the content of a document. This is crucial for various applications like document retrieval, search engine refinement, and creating summaries. While existing methods, often based on supervised machine learning, have shown promise, there's a continuous need to improve their accuracy and robustness. The challenge lies in effectively capturing the nuances of language and document structure to distinguish truly representative keywords from mere incidental terms, and to achieve performance comparable to or exceeding that of human indexers.
Methods
The paper builds upon an existing keyword extraction algorithm that uses three prediction models trained on documents with manually assigned keywords. The core of the work involves several refinements:
- NP-Chunk Extraction Refinement: The authors found that removing initial determiners (like "a," "an," "the") from noun phrase (NP) chunks significantly improved recall without a substantial drop in precision, leading to a better F-measure.
- General Corpus for Collection Frequency: Instead of calculating term frequency across the training/testing documents, a more general corpus (British National Corpus) was used. This provided a more robust measure of term rarity across a broader linguistic landscape, leading to improved performance.
- Optimized Positive Example Weights: Weights for positive examples were systematically adjusted to maximize the F-measure of the combined model on a validation set, rather than optimizing each classifier individually.
- Regression vs. Classification: The most significant methodological shift is treating the keyword extraction task as a regression problem instead of binary classification. This allows the models to output a continuous "keywordness" score for each candidate term, enabling a more nuanced ranking and selection process.
Datasets
- Training/Validation/Testing Data: The experiments utilize a dataset of 2,000 English scientific journal abstracts from the Inspec database. This set is divided into training (1,000 documents), validation (500 documents), and testing (500 documents). Each abstract has manually assigned keywords (uncontrolled terms) by professional indexers.
- General Corpus: The British National Corpus (BNC) is used for calculating collection frequency.
Key Insights and Ideas
- The "Keywordness" Continuum: Treating keyword extraction as a regression problem, rather than a binary classification, better reflects the reality that terms have varying degrees of relevance. This continuous scoring allows for more flexible ranking and selection.
- The Power of Generalization: Using a broader corpus (BNC) for collection frequency calculation provides a more stable and informative measure of term rarity, leading to better performance than relying solely on the specific document collection.
- Linguistic Nuances Matter: Removing determiners from NP-chunks highlights how even small linguistic adjustments can significantly impact performance, particularly recall.
- Consistency with Human Indexers: The improved system achieves an inter-judgement agreement of 28.2% with manually assigned keywords, which is comparable to the consistency observed among inexperienced professional indexers.
Key Results
- Removing initial determiners from NP-chunks increased the F-measure from 33.0 to 36.8.
- Using a general corpus (BNC) for collection frequency calculation improved the F-measure from 40.1 to 41.4.
- The regression-based approach achieved a higher F-measure (45.5) compared to the best classification run (41.6).
- Limiting the number of selected keywords to a maximum of twelve per document resulted in an F-measure of 44.0, with increased precision but decreased recall.
- The final combined model, using regression and other refinements, achieved an F-measure of 41.6 on the test set.
Limitations and Future Work
- Incomplete Keyword Coverage: The paper acknowledges that some documents may not receive any selected keywords, even with the regression approach. A proposed solution is to assign a minimum of three terms even if their regression value is below the threshold, which slightly impacts precision but improves recall and leaves the F-measure unaffected.
- Potential for Further NP-Chunk Refinement: The authors suggest that removing all initial determiners from NP-chunks, not just "a," "an," and "the," might yield even higher improvements.
- Subjectivity of Evaluation: The authors note that evaluating automatically extracted keywords is inherently challenging due to the subjective nature of keyword assignment, even among human indexers. While the system's consistency is comparable to inexperienced indexers, some automatically extracted keywords might not be considered by humans.
- Future Work: The paper implicitly suggests exploring more sophisticated linguistic features and potentially deeper semantic analysis to further enhance keyword extraction. The ability to rank keywords also opens avenues for applications that require a graded selection of terms.
Summary updated