Growing Multi-Domain Glossaries from a Few Seeds using Probabilistic Topic Models

Stefano Faralli, Roberto NavigliEMNLP 2013 · DOI

TL;DR

ProToDoG is a minimally supervised framework that builds massive, high-quality glossaries for dozens of domains starting from just five seed examples per topic. By combining web-scale bootstrapping with probabilistic topic models, it extracts thousands of accurate terms and definitions, significantly outperforming existing automated methods and manual resources like Wikipedia in domain-specific coverage.

What Problem Does This Paper Address?

Glossaries are essential for both human learning and Natural Language Processing (NLP) tasks like Question Answering and Word Sense Disambiguation. However, we face a "coverage gap": most online glossaries are small, and even massive aggregators like Google Define fail to categorize definitions by domain.

Existing automated approaches usually fall into two traps:

  1. Corpus-based methods require large, pre-collected domain-specific text collections, which are hard to find for niche subjects.
  2. Pattern-based methods (like searching for "X is a Y") often suffer from low recall because human language is too varied for a few rigid rules to capture.

The authors set out to create a system that can "grow" a glossary from scratch using the Web as its corpus, requiring almost no human intervention while maintaining high precision.

Methods

The authors introduce ProToDoG (Probabilistic Topics for multi-Domain Glossaries). The workflow operates in a recursive loop:

  1. Initial Seeding: A human provides ~5 "hypernym" pairs per domain (e.g., for Computer Science: compiler, computer program).
  2. Web Harvesting: The system queries search engines using these seeds plus the keyword "glossary." It then uses structural HTML patterns (like <b>term</b> – definition) to extract candidate (term, gloss) pairs from the results.
  3. Probabilistic Topic Modeling (PTM): This is the "secret sauce." Instead of simple keyword matching, ProToDoG uses Latent Dirichlet Allocation (LDA) to model domains as "topics." It builds a model based on previously accepted glossaries and uses it to calculate the probability that a new definition belongs to a specific domain. If a definition for "point" appears, the PTM helps decide if it belongs to Art (thickness) or Business (mortgage discount).
  4. Bootstrapping: The system automatically extracts new hypernyms from the definitions it just found and uses them as seeds for the next iteration.
  5. Gloss Recovery: At the very end, the system re-evaluates previously filtered-out definitions using the now-mature topic model, "rescuing" valid entries that were discarded when the model was still "learning" the domain.

Datasets

The researchers evaluated ProToDoG across 30 diverse domains (ranging from Arts to Warfare).

  • Gold Standards: They used 10 well-reputed, manually compiled online glossaries (e.g., the Reuters glossary for Finance, the Utah computing glossary for Computing) as benchmarks.
  • Baselines: They compared their results against a Bag-of-Words (BoW) approach, a heuristic-based extraction from Wikipedia, Google Define, and TaxoLearn (a state-of-the-art taxonomy learning framework).

Key Insights and Ideas

  • Joint Extraction is Key: Unlike systems that find terms first and then look for definitions, ProToDoG finds them together. This "jointness" ensures that the term and its context are linked from the start, reducing noise.
  • The Power of PTM over BoW: Simple word-overlap (BoW) filtering fails when domains share vocabulary. PTMs capture the underlying "gist" of a domain, allowing the system to distinguish between different senses of the same word based on the surrounding definitional text.
  • "Extra-Coverage": One of the most striking findings was that ProToDoG didn't just match the gold standards; it vastly exceeded them. It discovered thousands of valid technical terms that human experts had missed in the "official" glossaries.
  • Minimal Supervision: The fact that the system achieves 90%+ precision starting from only five pairs per domain demonstrates that the Web's inherent structure (like glossary pages) is a powerful, underutilized resource for structured knowledge.

Key Results

  • Precision: ProToDoG achieved remarkably high precision, with glosses rated between 89% and 99% accurate across different domains. This significantly outperformed baseline methods like Bag-of-Words and Wikipedia extraction in both accuracy and coverage.
Precision (P) and Coverage (C) metrics for PTM (ProToDoG) compared to BoW and Wiki baselines.
  • Scale: In the Business domain, while the gold standard had 1,777 terms, ProToDoG extracted 7,370 terms with 95% precision. Across all tested domains, the system consistently identified more terms and glosses than were present in the manually curated gold standards.
Comparison of term (t) and gloss (g) counts across domains for Gold standards, PTM (ProToDoG), BoW, and Wikipedia.
  • Comparison to Wikipedia: ProToDoG provided much better domain-specific coverage than Wikipedia. While Wikipedia is great for "Named Entities" (people, places), ProToDoG is superior at finding "Concepts" (technical terms and definitions).
  • Comparison to TaxoLearn: ProToDoG extracted an order of magnitude more terms and glosses than TaxoLearn while maintaining significantly higher precision.

Limitations and Future Work

  • Seed Sensitivity: While the authors didn't "fine-tune" their seeds, they acknowledge that the initial choice of seeds can influence the bootstrapping path.
  • Granularity: The current model doesn't distinguish between levels of specificity (e.g., it treats Biology and Science as flat topics rather than a hierarchy).
  • Language: The study focused on English. The authors suggest that by simply translating the "glossary" keyword, the system could be adapted to other languages, but this hasn't been tested yet.
  • Future Direction: The authors plan to explore "gloss-driven" taxonomy learning, using these extracted definitions to automatically build structured trees of knowledge.

Summary updated

BibTeX
@inproceedings{faralli2013growing,
  author = {Stefano Faralli and Roberto Navigli},
  title = {{Growing Multi-Domain Glossaries from a Few Seeds using Probabilistic Topic Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  doi = {10.18653/v1/d13-1018},
}
Figure