Abstract
Access to web-scale corpora is gradually bringing robust automatic knowledge base creation and extension within reach. To exploit these large unannotated—and extremely difficult to annotate—corpora, unsupervised machine learning methods are required. Probabilistic models of text have recently found some success as such a tool, but scalability remains an obstacle in their application, with standard approaches relying on sampling schemes that are known to be difficult to scale. In this report, we therefore present an empirical assessment of the sublinear time sparse stochastic variational inference (SSVI) scheme applied to RelLDA. We demonstrate that online inference leads to relatively strong qualitative results but also identify some of its pathologies—and those of the model—which will need to be overcome if SSVI is to be used for large-scale relation extraction.
Introduction
Building Knowledge Bases manually is an impossible task due to the infinite and ever-growing nature of potential relations. Unsupervised machine learning offers a natural solution, but existing probabilistic models like RelLDA typically rely on Gibbs sampling, which is difficult to scale due to its large memory footprint and requirement for multiple passes over the dataset. Furthermore, dependency parsing—a common feature in state-of-the-art models—presents a significant computational bottleneck at web scale. This paper explores scaling unsupervised relation extraction by applying Sparse Stochastic Variational Inference (SSVI) to RelLDA and utilizing a reduced feature set.
Methodology
The authors adapt the RelLDA generative model, where documents are treated as mixtures of relation clusters. To achieve scalability, two primary modifications were implemented:
- Sparse Stochastic Variational Inference (SSVI): Unlike batch methods, SSVI processes small minibatches and employs a sparse update rule, updating only the parameters for relations and features observed in the current batch.
- Feature Reduction: The model moves away from complex dependency trees, instead using entity surface strings, entity types, words between entities (grouped by POS), and POS tag sequences.
The study utilized the AQUAINT2 corpus, comprising approximately 578,790 documents and 1.5 million sentences, preprocessed with Stanford CoreNLP for POS tagging and NER.
Key Insights
A significant finding is that dependency parse features are not strictly necessary for discovering coherent relations, lowering the barrier for processing massive text collections. However, the authors noted a "topic trap" where relation clusters began to resemble general topics (e.g., "war") rather than specific relations (e.g., "attacked"). Additionally, including entity surface strings caused the model to cluster by entity identity rather than relationship type; removing these strings forced a focus on linguistic patterns.
Results: Efficiency and the Perplexity Paradox
The SSVI approach obtained strong qualitative results in a fraction of the time required for Gibbs sampling and with a significantly smaller memory footprint. The model successfully discovered distinct relation clusters, such as "occupies leadership position at" (e.g., Director General of) and "market strategist at."
However, a "perplexity paradox" was observed. Despite the efficiency of online inference, Gibbs sampling achieved better held-out perplexity for the same amount of computation. This is likely due to the inherent noise in stochastic gradient methods and suggests that SSVI might require more data or better-tuned learning rates to match the predictive accuracy of Gibbs sampling.
Limitations and Future Work
The model occasionally over-coarsens clusters, grouping unrelated concepts with similar syntactic patterns. Furthermore, the performance of online inference is highly sensitive to hyperparameters such as the learning rate and the number of relations (R).
Future work should incorporate "background" models to absorb non-relation words and prevent cluster pollution. Additionally, more robust, adaptive learning rate methods are needed to improve the stability of SSVI for this task.
Summary updated