When a modern language model processes the phrase "the President," it almost certainly maps that mention to the current head of state. But if that text was written in 1961, the correct referent is John F. Kennedy; if written in 1863, it is Abraham Lincoln.
Standard natural language processing models treat text as a static snapshot frozen in the present day. This "temporal blindness" creates severe vulnerabilities when models encounter historical archives, long-term document streams, or fast-evolving social media. Entities change roles, words shift meanings, and identical names recur across centuries. To resolve these ambiguities, a growing body of research focuses on integrating temporal context directly into language representations and entity resolution pipelines.
By treating time not as an isolated metadata tag but as a fundamental dimension of meaning, these techniques allow systems to dynamically shift their semantic space to match the era of the text.
The Failure of Static Representations
Why do standard models fail when moving across time horizons? The root cause is a combination of modern popularity bias and temporal ambiguity.
As Li et al. (2004) observed, entity disambiguation models frequently break down because they fail to utilize external situational context, such as the specific time period or geographic setting of a document. Without this context, a model defaults to the most globally frequent or modern interpretation of a term.
This issue manifests in two primary ways:
- Semantic Drift: Words acquire new meanings or connotations over time. D'Souza et al. demonstrate that standard language models struggle to detect these meaning shifts because they process text without explicit knowledge of when it was created, relying entirely on surrounding words to guess the era-specific sense.
- Entity Recurrence: Names and titles are frequently recycled. Honovich et al. point out that recurring names of monarchs, political titles, or historical events create massive ambiguity for machine reading systems. Without deep temporal context, a model cannot determine which "King George" or "Great Plague" a document is referencing.
Even outside of natural language, structural similarity alone is insufficient to resolve historical relationships. For instance, in network analysis, Serrà et al. show that purely structural or similarity-based connections cannot distinguish an original prototype from its later derivatives without integrating external temporal metadata like release dates.
Conditioning Language Models on Time
To build intuition for how to solve this, consider how humans read historical text: we mentally adjust our vocabulary and expectations based on the document's date. Researchers replicate this by explicitly conditioning language models on temporal metadata during training.
One straightforward yet highly effective mechanism is text prefixing. Pokrywka et al. appended publication dates directly as text prefixes to training data. This forces the model's internal attention mechanisms to condition its vocabulary predictions on the specific era, allowing it to naturally adapt to archaic linguistic contexts. Similarly, when dealing with centuries of historical records, Yoo et al. demonstrated that incorporating specific era or reign metadata as input features significantly boosts a model's performance in topic classification and named entity recognition.
Standard Input: "The economy experienced a severe panic..."
Conditioned Input: "[Era: 1893] The economy experienced a severe panic..."
Beyond simple prefixes, models can be forced to learn time as a continuous or granular variable through auxiliary training tasks. Jatowt et al. modified the pretraining objective to predict a document's specific publication month. This forces the model to map language shifts at a highly granular level, treating language as a continuously evolving entity.
When a model successfully internalizes this chronological progression, its predictive power increases. Li et al. (2024) showed that training language models on chronological document histories dramatically reduces uncertainty when predicting future content words. Even in short-form, modern communication like emails, Trajanovski et al. found that integrating time-based metadata remains a highly effective signal for predicting scheduling-related vocabulary when other historical context signals fail.
Time-Aware Entity Disambiguation
While language models use time to predict words, entity resolution systems use time to anchor ambiguous mentions to a precise real-world identity. The core intuition here is to align the text's creation time with the state of the world at that exact moment.
Ntarmos et al. proposed a framework that uses page timestamps to map mentions to a knowledge base as it existed at that specific point in time. Instead of searching a massive, modern knowledge base, the search space is restricted to the historical window of the document. Mishra et al. applied a similar paradigm to social media, linking generic or recurring references in tweets to specific temporal instances in Wikidata based on the tweet's creation time.
[Mention: "The Prime Minister"]
│
▼
┌──────────────┐ 1995 State ┌──────────────────┐
│ Document Date│ ───────────────────> │ John Major │
│ Oct 1995 │ └──────────────────┘
└──────────────┘ 2023 State ┌──────────────────┐
───────────────────> │ Rishi Sunak │
└──────────────────┘
To combat the bias where modern, highly popular entities overshadow historical ones, Agarwal et al. introduced "temporal signatures"—embeddings that represent an entity's relevance over time. By treating this temporal signature as a local context feature alongside the surrounding text, their system, diaNED, successfully resolves mentions that are otherwise indistinguishable to modern-biased models.
When explicit dates are missing from the text entirely, the relationship reverses: entities themselves can act as timestamps. Massidda demonstrated that while sentence embeddings capture general semantic style, specific named entities (like the names of short-lived treaties or politicians) provide the precise historical anchors needed for fine-grained document dating.
Once these entities and events are extracted, they can be organized sequentially. Barik et al. showed that mapping events onto a continuous chronological timeline allows models to evaluate the surrounding temporal context, making it possible to distinguish between multiple historical occurrences of the same event type.
Relative Temporal Binning
Not all temporal problems require knowing the exact calendar year. In localized contexts, such as medical records or legal proceedings, the critical factor is relative time—knowing whether an event happened before, during, or after a specific anchor point.
To solve this, Fosler-Lussier et al. introduced the concept of relative temporal binning for coreference resolution. In clinical notes, a patient might experience "coughing" across multiple hospital visits spanning several months. Treating every mention of "coughing" as the same entity instance would be a mistake.
Patient Timeline:
┌─────────────────────────────────┐ ┌─────────────────────────────────┐
│ Time-Bin A │ │ Time-Bin B │
│ (Admission Window) │ │ (Post-Op Follow-Up) │
├─────────────────────────────────┤ ├─────────────────────────────────┤
│ Mention 1: "patient's cough" │ │ Mention 3: "developed a cough" │
│ Mention 2: "the acute coughing" │ │ │
└─────────────────────────────────┘ └─────────────────────────────────┘
[Mentions 1 & 2 Corefer] [Mention 3 is a New Instance]
By mapping entity mentions into discrete time categories relative to a major event (like a surgery or hospital admission), the authors created a powerful discriminatory feature. Mentions that share high semantic similarity but fall into different relative temporal bins are statistically unlikely to refer to the same instance of a concept. This mechanism allows the system to segment a single recurring symptom into distinct, chronologically isolated medical episodes.
Summary
Whether operating across centuries of newspaper archives or days of clinical notes, time-aware text processing works because it constrains the hypothesis space. By injecting temporal metadata—via prefixes, auxiliary tasks, temporal signatures, or relative bins—systems stop viewing language as a flat, static plane and begin interacting with it as a dynamic, multi-layered history.
Key Foundational Work
Longitudinal Analytics on Web Archive Data: It's About Time! (2011) — contributed 2 concept elements to this direction.
Identification and Tracing of Ambiguous Names: Discriminative and Generative Approaches (2004) — contributed 1 concept elements to this direction.