Abstract
Searching historical web archives is difficult because language evolves; a query for "iPod" won't find 1980s articles about the "Walkman." This paper introduces a method to bridge this "terminology gap" by automatically reformulating modern queries into the language of the past using a Hidden Markov Model (HMM) and a novel measure of across-time semantic similarity.
Introduction
Information Retrieval (IR) systems generally rely on keyword matching. However, in long-term web archives (like the Internet Archive or newspaper digitizations), the terminology used by the author decades ago often differs from the terminology used by a modern searcher. The authors identify three main challenges:
- Temporal Drift: Names of entities change (e.g., Leningrad vs. Saint Petersburg) and technologies are replaced (e.g., Walkman vs. iPod).
- Contextual Mismatch: Standard query expansion fails because "iPod" and "Walkman" rarely appear in the same documents, meaning they have low statistical correlation in a traditional sense.
- Query Coherence: Simply replacing words one-by-one can lead to nonsensical queries (e.g., "Leningrad Smithsonian").
Methodology
Across-Time Semantic Similarity
To compare terms from different eras, the authors use a "contextual hypothesis." If term $u$ at time $T_1$ frequently co-occurs with the same set of words (e.g., "portable," "music," "earphones") that term $v$ co-occurs with at time $T_2$, then $u$ and $v$ are semantically similar across time.
HMM-based Query Reformulation
To ensure the reformulated query makes sense as a whole, they employ a Hidden Markov Model. The HMM balances three "desiderata":
- Similarity: How well does the reformulated term match the original term's meaning across time?
- Coherence: Do the terms in the reformulated query actually appear together in the target time period?
- Popularity: Does the reformulated term appear frequently enough in the archive to be a useful search term?
To make this computationally feasible, the authors use pruning and Top-k processing, employing the Viterbi algorithm combined with A* search to find the best overall reformulation.
Experimental Setup
The model was tested using the New York Times Annotated Corpus, containing over 1.8 million articles published between 1987 and 2007. Wikipedia was used as a "poor man's" entity extractor to identify multi-word expressions (like "John Lennon") as single tokens.
Key Results
The model successfully identified functional equivalents across time. For example, it mapped "mp3" (2005) to "audio cd" and "digital audio" (1990), and "Starbucks" (2005) to "Dunkin' Donuts" (1990), reflecting the dominant chains of the respective eras.
Despite the massive state space of a natural language vocabulary, pruning techniques allowed the system to generate reformulations in 3–7 seconds. While not as fast as a standard search engine, it is viable for specialized archival tools.
Conclusion and Future Work
This approach effectively bridges the terminology gap by treating context as a bridge and enforcing query coherence through an HMM. Future work includes moving beyond 1-to-1 term mapping and allowing for more fluid, ad-hoc time intervals rather than fixed calendar years.
Summary updated