Cross-referencing Using Fine-grained Topic Modeling

Jeffrey Lund, Piper Armstrong, Wilson Fearn, Stephen Cowley, Emily Hales, Kevin D. Seppi

Introduction

Cross-referencing—the process of linking related passages across a large corpus—is a cornerstone of deep reading and academic scholarship. However, creating these links manually is a monumental task. For instance, the cross-references in the LDS edition of the Bible required hundreds of volunteers working for seven years. This is fundamentally an $O(n^2)$ search problem: as the text grows, the number of potential connections increases quadratically. The goal of this research is to lower the "expertise barrier" by using machine learning to surface the most likely connections for human review.

An example of the cross-referencing interface showing related verses for Isaiah 25:8.

Methodology: Fine-Grained Topic Modeling

To address the search space problem, the system utilizes fine-grained topic modeling to generate candidate links for human verification. Unlike traditional "coarse" models that use roughly 100 topics, this approach utilizes 3,000 topics for a corpus of approximately 30,000 verses. This granularity ensures that each topic corresponds to a manageable number of documents, mirroring the density of human-curated references.

The technical approach leverages the Anchor Word algorithm, treated as a non-negative matrix factorization problem. A key innovation is Tandem Anchoring, where topics are seeded by the harmonic mean of word vectors within a randomly selected verse. This prevents the algorithm from selecting obscure or rare terms as anchors. Once trained, verses are represented as $K$-dimensional vectors, and similarity is calculated using various metrics, with cosine distance proving to be the most effective.

Evaluation and Results

The models were evaluated against three benchmarks: the Treasury of Scripture Knowledge, Enhanced (TSKE), and two subsets of crowdsourced data from OpenBible.info. To test generalizability, the model was also applied to the works of Jane Austen and Plato.

Precision-Recall curves comparing Tandem, Gram-Schmidt, and Coarse models across various distance metrics.

As shown in the performance metrics, fine-grained models (Tandem and Gram-Schmidt) significantly outperform traditional coarse models. In the Jane Austen corpus, 13% of the top 300 suggestions were valid thematic links, while in Plato's works, accuracy reached nearly 40%. Crucially, the fine-grained model identified "non-obvious" thematic links that simple keyword matching missed, such as connecting the "sting of a scorpion" to the "sting of death."

Discussion and Economic Impact

The primary value of this system is economic. By shifting the task from "expert search" to "novice verification," the cost of producing a comprehensive reference resource drops from millions of dollars in labor to approximately $7,500.

While the model is highly effective, it faces limitations with complex metaphors and redundant, nearly identical verses. Future work will focus on integrating semantic embeddings to better capture metaphorical relationships and refining anchor selection strategies to improve the discovery of subtle thematic connections.

Summary updated

Abstract

Cross-referencing, which links passages of text to other related passages, can be a valuable study aid for facilitating comprehension of a text. However, cross-referencing requires first, a comprehensive thematic knowledge of the entire corpus, and second, a focused search through the corpus specifically to find such useful connections. Due to this, cross-reference resources are prohibitively expensive and exist only for the most well-studied texts (e.g. religious texts). We develop a topic-based system for automatically producing candidate cross-references which can be easily verified by human annotators. Our system utilizes fine-grained topic modeling with thousands of highly nuanced and specific topics to identify verse pairs which are topically related. We demonstrate that our system can be cost effective compared to having annotators acquire the expertise necessary to produce cross-reference resources unaided.
BibTeX
@inproceedings{lund2019cross,
  author = {Jeffrey Lund and Piper Armstrong and Wilson Fearn and Stephen Cowley and Emily Hales and Kevin D. Seppi},
  title = {{Cross-referencing Using Fine-grained Topic Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1399},
}
Figure