Bridging the Terminology Gap in Web Archive Search

Klaus Berberich, Srikanta J. Bedathur, Mauro Sozio, Gerhard WeikumWebDB 2009 · OpenReview

Abstract

Searching historical web archives is difficult because language evolves; a query for "iPod" won't find 1980s articles about the "Walkman." This paper introduces a method to bridge this "terminology gap" by automatically reformulating modern queries into the language of the past using a Hidden Markov Model (HMM) and a novel measure of across-time semantic similarity.

Introduction

Information Retrieval (IR) systems generally rely on keyword matching. However, in long-term web archives (like the Internet Archive or newspaper digitizations), the terminology used by the author decades ago often differs from the terminology used by a modern searcher. The authors identify three main challenges:

  1. Temporal Drift: Names of entities change (e.g., Leningrad vs. Saint Petersburg) and technologies are replaced (e.g., Walkman vs. iPod).
  2. Contextual Mismatch: Standard query expansion fails because "iPod" and "Walkman" rarely appear in the same documents, meaning they have low statistical correlation in a traditional sense.
  3. Query Coherence: Simply replacing words one-by-one can lead to nonsensical queries (e.g., "Leningrad Smithsonian").

Methodology

Across-Time Semantic Similarity

To compare terms from different eras, the authors use a "contextual hypothesis." If term $u$ at time $T_1$ frequently co-occurs with the same set of words (e.g., "portable," "music," "earphones") that term $v$ co-occurs with at time $T_2$, then $u$ and $v$ are semantically similar across time.

Figure 1: Illustration of across-time semantic similarity using bridge terms to link 'iPod' (2005) and 'Walkman' (1990).

HMM-based Query Reformulation

To ensure the reformulated query makes sense as a whole, they employ a Hidden Markov Model. The HMM balances three "desiderata":

  • Similarity: How well does the reformulated term match the original term's meaning across time?
  • Coherence: Do the terms in the reformulated query actually appear together in the target time period?
  • Popularity: Does the reformulated term appear frequently enough in the archive to be a useful search term?

To make this computationally feasible, the authors use pruning and Top-k processing, employing the Viterbi algorithm combined with A* search to find the best overall reformulation.

Experimental Setup

The model was tested using the New York Times Annotated Corpus, containing over 1.8 million articles published between 1987 and 2007. Wikipedia was used as a "poor man's" entity extractor to identify multi-word expressions (like "John Lennon") as single tokens.

Key Results

The model successfully identified functional equivalents across time. For example, it mapped "mp3" (2005) to "audio cd" and "digital audio" (1990), and "Starbucks" (2005) to "Dunkin' Donuts" (1990), reflecting the dominant chains of the respective eras.

Figure 2: Top-10 reformulations for single-term queries across different time slices.

Despite the massive state space of a natural language vocabulary, pruning techniques allowed the system to generate reformulations in 3–7 seconds. While not as fast as a standard search engine, it is viable for specialized archival tools.

Conclusion and Future Work

This approach effectively bridges the terminology gap by treating context as a bridge and enforcing query coherence through an HMM. Future work includes moving beyond 1-to-1 term mapping and allowing for more fluid, ad-hoc time intervals rather than fixed calendar years.

Summary updated

BibTeX
@inproceedings{berberich2009bridging,
  author = {Klaus Berberich and Srikanta J. Bedathur and Mauro Sozio and Gerhard Weikum},
  title = {{Bridging the Terminology Gap in Web Archive Search}},
  booktitle = {WebDB},
  year = {2009},
  url = {https://openreview.net/forum?id=0NoaH1-vnLF7},
}
Figure