Learning the Similarity of Documents: An Information-Geometric Approach to Document Retrieval and Categorization

Thomas HofmannNeurIPS 1999 · OpenReview

This paper, authored by Thomas Hofmann, presents a mathematically rigorous way to measure document similarity by blending generative modeling with information geometry. It moves beyond simple word-counting to a framework that can automatically account for synonyms and words with multiple meanings.

TL;DR

The paper introduces a method for learning document similarity by deriving a Fisher kernel from a latent class generative model (pLSA). By treating documents as points on a curved statistical manifold, the author creates a similarity metric that significantly outperforms traditional methods in both information retrieval and text categorization, especially when labeled data is scarce.

What Problem Does This Paper Address?

In the late 90s, the "Vector Space Model" (VSM) was the standard for document retrieval. VSM treats documents as vectors of word counts, but it has two major flaws:

  1. Synonymy: If one document uses the word "automobile" and another uses "car," VSM sees them as completely unrelated.
  2. Polysemy: If the word "bank" appears in a financial document and a geography document, VSM treats them as similar because they share a word, even though the meanings differ.

The challenge was to create a similarity measure that is "data-driven"—learning these relationships from the text itself without needing a human-coded thesaurus—and "principled," meaning it's derived from statistical theory rather than just heuristics.

Methods

The author’s approach involves three main steps:

  1. Latent Class Decomposition: Also known as Probabilistic Latent Semantic Analysis (pLSA), this unsupervised method models each document as a mixture of "latent classes" or topics. Instead of just looking at words, we look at the probability that a document belongs to certain topics.
Examples of latent classes learned from text
  1. The Fisher Kernel: This is the core innovation. A Fisher kernel measures how much the parameters of a generative model (like the topic model above) would need to change to better represent a specific document. This "gradient" captures the unique signature of a document relative to the entire collection.
  2. Information Geometry: To make the math tractable, the author uses a "square-root parameterization." This maps the documents onto a hypersphere, allowing the complex "Fisher information matrix" to be approximated by a simple identity matrix. This makes computing similarity as easy as taking a weighted inner product.

Datasets

The paper evaluates the method on several classic benchmarks:

  • Information Retrieval: Medline, Cranfield, CACM, and CISI (standard collections of abstracts and articles).
  • Text Categorization: The Reuters-21578 corpus, a collection of news stories categorized by topics like "earn," "grain," and "crude." The author specifically tested "subsampled" versions of this dataset to see how the model performs with very little training data.

Key Insights and Ideas

  • Capturing Context: The Fisher kernel naturally handles polysemy. For example, the word "president" might appear in a "corporate" topic and a "politics" topic. The model looks at the context of the document to decide which "president" is being discussed, preventing false matches.
  • The Power of Unlabeled Data: One of the biggest "aha" moments is that you can train the generative model on a massive pile of unlabeled documents to learn the structure of language. You can then use the resulting similarity kernel to perform highly accurate classification with only a handful of labeled examples.
  • Geometric Similarity: By viewing documents through the lens of information geometry, similarity isn't just a score—it's a distance on a curved manifold that represents the space of all possible word distributions.

Key Results

  • Retrieval Boost: On the Medline dataset, the Fisher kernel approach (dubbed VSM++) achieved an average precision of 67.2%, compared to just 44.3% for the standard Vector Space Model.
  • Small-Sample Success: In text categorization, the method showed its greatest strength when data was limited. When training on only 1/20th of the Reuters data, the Fisher kernel improved SVM performance by 27.4% and k-Nearest Neighbor performance by 14.7%.
Text categorization performance on the Reuters dataset across different subsampling levels
  • Diminishing Returns with Scale: Interestingly, for Support Vector Machines (SVMs), the advantage of the Fisher kernel disappeared when the full dataset was available. This suggests that with enough labeled data, a standard linear SVM can eventually "learn" what the generative model already knew.

Limitations and Future Work

  • The "Bag-of-Words" Constraint: The model assumes documents are "memoryless" sources, meaning it ignores word order. The author notes that extensions to more complex language models are possible but were beyond the scope of this initial paper.
  • Computational Cost: Training latent class models and computing gradients for every document pair is significantly more computationally expensive than simple VSM, which might limit its use in real-time systems with billions of documents.
  • Parameter Sensitivity: The choice of $K$ (the number of latent topics) is crucial. The author suggests combining models of different "resolutions" (different $K$ values) to get the best results, adding another layer of complexity to the implementation.

Summary updated

Abstract

The project pursued in this paper is to develop from first information-geometric principles a general method for learning the similarity between text documents. Each individual docu(cid:173) ment is modeled as a memoryless information source. Based on a latent class decomposition of the term-document matrix, a low(cid:173) dimensional (curved) multinomial subfamily is learned. From this model a canonical similarity function - known as the Fisher kernel - is derived. Our approach can be applied for unsupervised and supervised learning problems alike. This in particular covers inter(cid:173) esting cases where both, labeled and unlabeled data are available. Experiments in automated indexing and text categorization verify the advantages of the proposed method.
BibTeX
@inproceedings{hofmann1999learning,
  author = {Thomas Hofmann},
  title = {{Learning the Similarity of Documents: An Information-Geometric Approach to Document Retrieval and Categorization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1999},
  url = {https://openreview.net/forum?id=rkZ5yuWO-H},
}
Figure