Cross-referencing using Fine-grained Topic Modeling

Jeffrey Lund, Piper Armstrong, Wilson Fearn, Stephen Cowley, Emily Hales, Kevin Seppi

TL;DR

Creating cross-references for large bodies of text is a labor-intensive $O(n^2)$ problem that traditionally requires years of expert study. This paper introduces a system that uses fine-grained topic modeling (thousands of topics) to automatically suggest candidate cross-references, demonstrating that AI-assisted curation can reduce the cost of building these resources by orders of magnitude.

What Problem Does This Paper Address?

Cross-references—links between related passages—are essential for deep comprehension of complex texts like legal codes, religious scriptures, or academic textbooks. However, they are prohibitively expensive to create. Unlike part-of-speech tagging, where a human can look at a single sentence in isolation, cross-referencing requires an annotator to have a comprehensive thematic map of the entire corpus in their head to spot connections.

Because the number of potential pairs grows quadratically with the number of passages, exhaustive manual cross-referencing is rare. For example, the cross-references in the LDS edition of the Bible took hundreds of volunteers seven years to produce. The authors aim to bridge this gap by building a system that identifies topically related passages even when they don't share specific keywords, acting as a "recommendation engine" for human annotators.

Methods

The authors move beyond simple keyword matching (concordances) by utilizing fine-grained topic modeling.

  1. High Topic Count: While standard Latent Dirichlet Allocation (LDA) models typically use ~100 topics, this system uses 3,000 topics to capture highly nuanced themes.
  2. Anchor Word Algorithm: To handle such a high number of topics efficiently, they use the Anchor Word algorithm, which treats topic modeling as a non-negative matrix factorization problem.
  3. Tandem Anchoring: A novel contribution is how they "seed" these topics. Instead of using the default Gram-Schmidt process (which often picks esoteric words), they use tandem anchors. They randomly select a verse, take its words, and calculate their element-wise harmonic mean in a high-dimensional word-vector space to form a single topic anchor.
  4. Vector Comparison: Once verses are assigned topic distributions (represented as $K$-dimensional vectors), the system compares them using distance metrics. They tested Cosine, Euclidean, Manhattan, and Chebyshev distances to find which best identifies valid links.

Datasets

The primary evaluation was conducted on the English Standard Version (ESV) of the Bible, chosen because it is one of the few texts with extensive "ground truth" cross-reference data:

  • TSKE: The "Treasury of Scripture Knowledge, Enhanced," containing ~670,000 human-curated links.
  • OpenBible.info: A crowdsourced dataset where users vote on the helpfulness of specific cross-references (subsets used: OpenBible+0 and OpenBible+5 based on net votes).

To test generalization, they also applied the model to the complete works of Jane Austen and Plato.

Key Insights and Ideas

  • Fine-Grained is Better: The jump from 100 to 3,000 topics was crucial. Coarse models are too broad to distinguish specific cross-references; fine-grained models provide the "resolution" needed to link specific ideas.
  • Cosine Distance Wins: For fine-grained models, cosine distance consistently outperformed other metrics in identifying valid cross-references, as shown in their ROC and Precision-Recall curves.
  • Thematic vs. Keyword Links: The system successfully identifies "thematic" links that share no significant words. For example, it linked a passage in Austen's Pride and Prejudice to one in Emma because both discussed the intersection of social status and marriage, despite having no overlapping vocabulary.
  • Explicit vs. Implicit Ideas: The model performed significantly better on Plato (119/300 valid links) than Jane Austen (39/300). The authors suggest this is because Plato discusses ideas explicitly (using words like "virtue" or "justice"), whereas Austen conveys themes implicitly through character interaction, requiring the model to "read between the lines."

Key Results

The system's performance was evaluated using Receiver Operating Characteristic (ROC) and Precision-Recall Curves (PRC).

Figure 2: ROC curves showing that fine-grained models (Tandem/Gram-Schmidt) significantly outperform the coarse baseline across different metrics.
  • Cost Efficiency: The authors estimate that to produce a resource of 12,000 valid cross-references, a naive manual search would cost roughly $48,000,000**. Using their fine-grained model to suggest candidates for human verification drops that cost to approximately **$7,500.
  • Metric Performance: As seen in Figure 3, the precision of the fine-grained models is significantly higher than the coarse baseline, especially at lower recall levels.
Figure 3: PRC plots illustrating that cosine distance with fine-grained topics yields the best precision for cross-reference selection.

Limitations and Future Work

  • The "Identical Verse" Trap: The model often flags verses that are nearly identical (e.g., "The Lord said to me...") as high-confidence links. While technically related, these are often considered "low-value" by human editors.
  • Metaphor and Context: The system struggles with metaphorical connections or links that require deep cultural context (e.g., linking descriptions of the biblical tabernacle).
  • Implicit Themes: As noted with the Jane Austen dataset, capturing themes that are never explicitly named remains a challenge for current topic modeling approaches.
  • Future Directions: The authors suggest exploring fine-grained topic modeling for other tasks requiring high precision and nuance, beyond just cross-referencing.

Summary updated

Abstract

Cross-referencing, which links passages of text to other related passages, can be a valuable study aid for facilitating comprehension of a text. However, cross-referencing requires first, a comprehensive thematic knowledge of the entire corpus, and second, a focused search through the corpus specifically to find such useful connections. Due to this, cross-reference resources are prohibitively expensive and exist only for the most well-studied texts (e.g. religious texts). We develop a topic-based system for automatically producing candidate cross-references which can be easily verified by human annotators. Our system utilizes fine-grained topic modeling with thousands of highly nuanced and specific topics to identify verse pairs which are topically related. We demonstrate that our system can be cost effective compared to having annotators acquire the expertise necessary to produce cross-reference resources unaided.
BibTeX
@inproceedings{lund2019cross,
  author = {Jeffrey Lund and Piper Armstrong and Wilson Fearn and Stephen Cowley and Emily Hales and Kevin Seppi},
  title = {{Cross-referencing using Fine-grained Topic Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1399},
}
Figure