Keyword Extraction Using Unsupervised Learning on the Document’s Adjacency Matrix

Eirini Papagiannopoulou, Grigorios Tsoumakas, Apostolos PapadopoulosNAACL 2021

TL;DR

This paper introduces a novel unsupervised approach to keyword extraction by modeling the distribution of a document's "adjacency matrix" (representing word co-occurrences). By treating this matrix as a vector representation, they leverage unsupervised learning algorithms to identify important keywords, achieving competitive results with state-of-the-art methods without requiring labeled data.

What Problem Does This Paper Address?

Automatic Keyword Extraction (AKE) aims to automatically identify the most representative words or phrases in a document. While many methods exist, recent advancements have leaned heavily towards supervised deep learning, showing diminishing returns for traditional unsupervised methods. This paper addresses the stagnation of unsupervised AKE, particularly graph-based approaches, by proposing a new way to leverage the information within the graph-of-words. Existing graph-based methods often rely on centrality measures like PageRank, but this paper argues that the distribution of the adjacency matrix itself holds valuable information that has been underutilized. Furthermore, many methods arbitrarily fix the number of keywords, rather than letting the document's content dictate it.

Methods

The core of this work lies in treating the adjacency matrix of a document's graph-of-words as a vector representation of its vocabulary. They explore three versions of this adjacency matrix:

  • Unweighted: Simple co-occurrence (1 if words co-occur within a window, 0 otherwise).
  • Weighted: Co-occurrence count within a window.
  • Weighted with Positional Information: Incorporates the first sentence a word appears in, aiming to capture more nuanced relationships.

The novelty lies in applying unsupervised learning algorithms to model the distribution of these adjacency matrices. Instead of just using centrality, they explore:

  • Mean Vector Approach: Calculating the mean of the word vectors (rows of the adjacency matrix) and scoring words based on their Euclidean distance from this mean. Words further away are considered more important.
  • One-Class SVM: Using a hyperplane to separate "inlier" (common, less important) words from "outlier" (important, keyword) words. Words with a negative score (further from the inlier region) are ranked higher.
  • Isolation Forest: Building an ensemble of isolation trees to identify words that are easily "isolated" from the rest, implying they are distinct and thus likely keywords. This method naturally determines the number of keywords.

Datasets

The authors evaluated their methods on six publicly available datasets:

  • Full-text publications: NUS, SemEval (SE), and ACM.
  • Scientific abstracts: KDD and WWW.
  • News texts: DUC-2001.

They describe specific strategies for splitting these datasets into training and testing sets, noting that guidelines are not always clear for all datasets.

Key Insights and Ideas

  • Adjacency Matrix as a Rich Representation: The paper convincingly argues that the adjacency matrix, when viewed as a collection of word vectors, contains more information than just pairwise connections. Its distribution is key.
  • Distribution Matters for Unsupervised AKE: By modeling the distribution of these word vectors, they can effectively distinguish keywords from non-keywords without needing labeled data.
  • Positional Information Enhances Performance: Incorporating word position into the adjacency matrix (the 'Q' matrix) generally leads to improved keyword extraction performance, especially for distribution-based methods.
  • Isolation Forest Naturally Determines Keyword Count: The Isolation Forest approach offers a significant advantage by inherently determining the number of keywords based on the data's structure, avoiding arbitrary parameter settings.
  • Keywords are "Outliers": The core intuition behind the SVM and Isolation Forest approaches is that keywords are anomalies or outliers compared to the vast majority of common, less relevant words in a document.

Key Results

  • The Isolation Forest (IF) approach, particularly with the positional information matrix (IFQ), demonstrated the best effectiveness in terms of F1 scores across most datasets.
  • The paper shows that incorporating more information (weighted edges, then positional information) generally leads to higher F1 scores for most methods, with notable exceptions.
  • Spearman's correlation coefficients revealed a strong positive correlation (≥ 0.8) between the IF scoring function and traditional graph-based methods like PageRank (PA) and Node Degree (NA), suggesting a shared underlying principle.
  • The mean vector approach (LV) also achieved competitive results, particularly when using the positional matrix (LVQ).
  • The One-Class SVM (OC) approach showed significant improvements when using the positional matrix (OCQ).

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while their methods perform well, they are not always statistically significantly better than all competitive approaches, especially when comparing different types of graph inputs.
  • The performance of the One-Class SVM and Isolation Forest on news datasets (DUC) was noted as needing further investigation.
  • Future Work:
    • Improving the scoring functions of unsupervised learning algorithms.
  • Adapting the proposed approach to keyphrase extraction (not just keywords).
  • Developing novel distribution-based modeling methods that can simultaneously utilize information from multiple adjacency matrices.
  • Applying this adjacency matrix distribution modeling to other tasks that currently rely solely on graph-based methods.

Summary updated

BibTeX
@inproceedings{papagiannopoulou2021keyword,
  author = {Eirini Papagiannopoulou and Grigorios Tsoumakas and Apostolos Papadopoulos},
  title = {{Keyword Extraction Using Unsupervised Learning on the Document’s Adjacency Matrix}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure