Distant supervision is a cheap way to build training data. It labels a sentence with a relation whenever the sentence mentions two entities that a knowledge base links. The field's standard complaint is that co-occurrence does not imply relation. This proposal targets a second, largely unmeasured failure: the knowledge base describes the world at one time, and the sentence was written at another. A 2015 article about a company's chief executive, aligned against a 2024 knowledge base, gets confidently mislabeled. The error is not random. It grows with the time gap between document and knowledge source, it concentrates in relations that change often (office holders, employers, spouses, team membership), and the offending sentence often does express a relation, only one that was true at a different time. Denoisers built to catch sentences that "don't really say the relation" have no reason to catch these.
The same problem reappears, disguised, when a large language model (LLM) replaces the knowledge base as the labeler. The LLM's knowledge base is its frozen training data, so its "snapshot time" is its training cutoff. Its labels are then checked on benchmarks like CoNLL03 that it has very likely memorized, so the stale-knowledge component of its noise is invisible. The claim of this note is that temporal mismatch is a measurable, separable part of distant supervision noise. Knowing the time gap, together with a predicted lifespan for each fact, lets a denoiser remove it better than time-blind methods. Evaluation on text written after the labeler's cutoff exposes LLM-labeler noise that contaminated benchmarks hide.
Core Claim
We make three testable claims. First, in distantly supervised relation extraction over dated news, the label error rate rises monotonically with the gap between the document date and the validity interval of the aligned fact. This rise is steepest for volatile relations. Second, a noise model that turns each silver label into a soft target is enough to beat strong time-agnostic denoisers on held-out, human-annotated, time-sliced test data. The soft target is the probability that the fact was valid on the document date, estimated from knowledge-base intervals when they exist and from a learned fact-lifespan predictor when they do not. Third, when an LLM serves as the distant labeler, its agreement with gold labels drops on post-cutoff text in a specific direction: it asserts the previous holder of a relation. That drop is hidden on pre-cutoff benchmarks.
The mechanism is a single weighting rule. For a silver label linking entities $s,o$ by relation $r$ in a document dated $t_{doc}$, with a knowledge-base start time $t_{start}$ and predicted lifespan $d$, we set the soft target to
$$w = \Pr\left[\, D > t_{doc} - t_{start} \,\right], \quad D \sim p_\theta(\cdot \mid s, r, o),$$
Here $D$ is the random lifespan of the fact, and $p_\theta$ is a lifespan distribution predicted from the fact's text. When the knowledge base supplies an end time, $w$ collapses to an indicator of whether $t_{doc}$ falls inside the interval. For documents dated before $t_{start}$, $w$ is the analogous probability computed in reverse. The rule deliberately uses the start time: the file notes that existing lifespan-based calibration ignores it.
Relation to Prior Work
Two papers motivate the gap. Mitigating Temporal Misalignment by Discarding Outdated Facts trains a BERT fact-duration predictor on distantly supervised dated news (TimeQA, TimePre). It compares the predicted lifespan to the gap between query time and evidence time, and uses the comparison to lower QA confidence or trigger retrieval, reporting a 50–60% reduction in expected calibration error on SituatedQA. We reuse its duration predictor as our $p_\theta$. We point it the other way: at training labels rather than test-time answers, and we add the start time it omits. RealTime QA documents the "outdated trap": when retrieved context is insufficient, models fall back on stale training knowledge. We hypothesize that the same fallback governs LLM labelers facing ambiguous post-cutoff sentences, and we test this directly.
The denoising methods we compete with treat noise as time-independent. Debiased and Denoised Entity Recognition from Distant Supervision (DesERT) uses a two-head architecture, peer networks that accept a pseudo-label only when both agree, and a worst-case loss. It also shows that ChatGPT-generated silver labels carry structural bias. However, it evaluates only on CoNLL03, OntoNotes, Webpage, Wikigold and Twitter, all old benchmarks. The Semi-Supervised Teacher-Student Architecture for Relation Extraction enforces agreement between a student and a slowly updated copy of itself. Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation replaces hard targets with soft ones, and we borrow its soft-target training. None of these condition on time, and we predict they will be weakest on exactly the stale-but-well-formed sentences described above. Position-aware Attention and Supervised Data Improve Slot Filling argued that data quality caps relation extraction and built TACRED from human annotation. We use it as the architecture reference and as a pre-cutoff clean point.
The evaluation design is constrained by the contamination literature. Test of Time shows that post-cutoff performance drops can vanish when questions are rephrased, even though the memorized advantage persists. That warns us not to read a post-cutoff drop in LLM-labeler accuracy as proof of anything unless difficulty and phrasing are controlled. Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? builds a post-cutoff news set from BBC and The Guardian and checks freshness with Min-K% Prob, a membership-inference score. We adopt both the sources and that check. Temporal knowledge editing work, History Matters and TEMPLE-MQA, assumes clean timed triples from YAGO. Our output, denoised timed triples from raw text, is what those methods would need in practice. We treat that connection as a downstream check, not the core contribution.
Research Program
The first component is a dated distant supervision corpus. We collect news with publication dates spanning roughly a decade, drawing on the outlets used by RealTime QA and CausalProbe-2024. Entity pairs are aligned against the timed triples of YAGO3.0.34, the same source AToKe uses. We perform the alignment twice: once against a single present-day snapshot, which is standard practice, and once ignoring time entirely. We then have annotators label a stratified test sample, stratified by relation volatility and by time gap, so that each test sentence carries gold relations valid on its publication date. The annotation guideline follows TACRED's relation inventory where it overlaps. The sample also includes an explicit post-cutoff slice for each LLM labeler studied, with Min-K% Prob used to confirm the slice is fresh.
The second component is measurement. Using the gold sample, we decompose silver-label errors into three types. The first is classic co-occurrence noise, where the sentence never expresses the relation. The second is temporal mismatch, where the sentence expresses a relation that held at a different time than the knowledge base records. The third is other error. We plot the temporal share as a function of gap and volatility. This decomposition is itself a result the field lacks.
The third component is the time-aware denoiser. It is the lifespan-weighted soft target above, trained with a standard position-aware relation classifier. We also combine it with DesERT's agreement filter to test whether temporal weighting and time-agnostic denoising are complementary.
The fourth component is the LLM-labeler study. We prompt LLMs zero-shot to label the same sentences before and after their cutoffs. We record whether errors on post-cutoff text name a relation's earlier holder, which is the signature of stale knowledge overriding the text. To separate staleness from difficulty, we build a matched control. We swap the entities in each post-cutoff sentence for invented names that the model cannot have prior beliefs about, keeping the syntax fixed. If accuracy recovers on the invented-name version, the loss came from stale knowledge rather than harder sentences.
Baselines
The naive reference point is raw distant supervision: a position-aware relation classifier trained on hard labels from present-snapshot alignment, with no denoising. On TACRED this architecture reaches 65.4% F1 (Position-aware Attention and Supervised Data Improve Slot Filling). That figure is a clean-supervision ceiling on old text, not a number on our corpus, and no reported result exists on Lacuna for any method on a dated, time-sliced distant supervision test set.
The strongest simple competitor is interval matching. Here a label is kept only if the document date falls inside the fact's YAGO validity interval, and discarded otherwise. It is the obvious fix and it will do well on relations with complete intervals. Our method must match it there and beat it on the large share of facts with missing end dates, where interval matching must either keep or drop everything.
The strongest existing denoiser is DesERT (Debiased and Denoised Entity Recognition from Distant Supervision). It reports +3.26% F1 on CoNLL03 and +4.26% on Webpage over prior distant supervision methods for entity recognition. We adapt its agreement filter and worst-case loss to relation classification. The teacher-student consistency method (Semi-Supervised Teacher-Student Architecture for Relation Extraction) serves as a second time-agnostic denoiser. A zero-shot LLM labeler, with its outputs used as training labels, is the fourth baseline.
All systems share the same silver corpus, encoder, training steps and hyperparameter search budget, and are selected on a small dated development set. Beating a baseline means higher macro-F1 on the time-stratified gold test set, with the gain concentrated in the large-gap, high-volatility buckets. Bootstrap confidence intervals must exclude zero.
The decisive ablation keeps our soft-target training but shuffles the time gaps across examples. This preserves the distribution of weights while destroying their link to time. If the shuffled version matches the real one, the gain comes from soft labels in general, not from temporal information, and the core claim fails.
Why not just…?
Why not just filter labels by the knowledge base's validity intervals? This is a real threat and we treat it as the main baseline rather than a straw man. It fails only where intervals are incomplete or absent, and where no knowledge base exists at all, as with LLM labelers. The decomposition experiment reports how much of the corpus falls in those regions. If interval matching captures nearly all of the temporal error, the correct conclusion is that the lifespan predictor adds little. We would then report the error decomposition and the LLM-labeler finding as the contribution.
Why not just use a strong time-agnostic denoiser like DesERT? Such denoisers look for labels that are inconsistent with what the sentence says. A stale label frequently sits on a sentence that fluently states a relation, just for the wrong time. We expect these methods to leave the large-gap buckets largely untouched. The per-bucket comparison, and the combination of DesERT with temporal weighting, will show whether the two remove different errors.
Why not just let a current LLM label the data? DesERT's own experiments show that ChatGPT silver labels carry structural bias. RealTime QA shows that models revert to stale knowledge when context is thin. Moreover, labeler quality is usually judged on benchmarks the labeler has likely seen. Our post-cutoff slice with invented-name controls measures this directly. If LLM labels show no stale-holder errors after the cutoff, this alternative wins for fresh text, and we will say so.
Why not just trust a post-cutoff split to reveal the LLM labeler's true accuracy? Test of Time shows that temporal drops can be hidden by rephrasing, and they can also be caused by harder post-cutoff material. For that reason we pair every post-cutoff sentence with its invented-name counterpart and verify freshness with Min-K% Prob, rather than relying on the date split alone.
Proposed Experiments
The first decisive check is the error decomposition. We ask whether silver-label error rises with the time gap, with a steeper slope for volatile relations than for stable ones such as birthplace. A flat curve would falsify the premise.
The second check is the main comparison. Using the same encoder and budget throughout, we compare raw distant supervision, interval matching, DesERT, teacher-student consistency, LLM labels, our lifespan-weighted targets, and the shuffled-gap ablation, reporting macro-F1 per gap-by-volatility bucket on the human-annotated dated test set. We also report TACRED F1 to confirm that nothing regresses on old, clean text.
The third check is the LLM-labeler study. We measure pre-cutoff versus post-cutoff agreement, the fraction of post-cutoff errors that name an earlier holder, and recovery under invented entity names. We add the same comparison on CoNLL03 versus fresh entity text to test whether the benchmark hides the effect.
As a secondary downstream check, we load denoised triples from post-cutoff news as the external memory of a time-aware retrieval graph in the style of TEMPLE-MQA. We then compare the recall of both current and historical facts against memory built from raw or interval-matched triples, and use this to test whether cleaner extraction carries through to question answering.
References
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- RealTime QA: What's the Answer Right Now?
- Debiased and Denoised Entity Recognition from Distant Supervision
- Semi-Supervised Teacher-Student Architecture for Relation Extraction
- Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation
- Position-aware Attention and Supervised Data Improve Slot Filling
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- History Matters: Temporal Knowledge Editing in Large Language Model
- Multi-hop Question Answering under Temporal Knowledge Editing