Large language models used as automatic graders ("LLM judges") are increasingly asked whether two texts in different languages describe the same thing. Examples include whether a Hindi news report and an English one cover the same event, or whether a name written in Chinese script is the same person as an English name. These judges are unreliable outside high-resource languages. Across 25 languages their verdicts agree with each other only at a Fleiss' Kappa of about 0.3 (How Reliable is Multilingual LLM-as-a-Judge?), and they prefer machine-translated text for reasons unrelated to content (DIBJudge). An older line of work offers evidence that does not depend on reading the language at all. When a real event happens, mentions of it surge at the same moments in independent news streams in every language. Matching the shape over time of those surges, called temporal burst correlation, can pair entities across languages with no dictionary or parallel text (Klementiev et al., Tao et al.).
This note proposes treating the burst signal as a second, independent witness that is combined with the judge outside the judge's prompt. The intuition is that the two sources fail for different reasons. The judge fails when its grasp of a language is weak. The burst signal fails when two unrelated things happen to spike in the same week. Because the failures differ, a small calibrated combination should repair judge verdicts exactly where the judge is weakest. The proposal also tests the obvious alternative, pasting the time series into the judge's prompt, and predicts that it will work worse.
Core Claim
The task is a cross-lingual correspondence judgment: given two items in different languages, decide whether they refer to the same real-world event or entity. We claim that a late-fused verdict will be more accurate and more consistent across languages than the judge alone, with the largest gains in low-resource languages. "Late-fused" means the judge's own confidence plus three or four burst-shape features, fed to a tiny calibrated classifier trained only on high-resource language pairs. We also claim the gain comes from the shape of the bursts, not merely from knowing that two items were published close together. Our final claim is that giving the judge the same burst information as text in its prompt captures little of this gain. Two prior findings motivate that last prediction. LLMs handle magnitude spikes but have a "variance blind spot" on time series (TSQueryBench). They also act on cues without acknowledging them in their explanations (The Silent Judge).
The closest existing work differs as follows. TSQueryBench asks judges to evaluate explanations of a single synthetic series and never compares two real streams. GlobeSumm groups multilingual news by event and lets GPT-4 resolve conflicts, but it takes event grouping from metadata and uses no temporal signal. DIBJudge repairs multilingual judges by retraining their internal representations. We leave the judge untouched and supply external evidence instead.
Research Program
The burst features come from the classic toolkit, and we reuse each method as published. Each candidate item gets a daily mention-count series in its own language's stream, drawn from per-language news counts in GDELT (the event source behind GlobeSumm) and Wikipedia edition activity. From each pair of series we compute three features. The first is the similarity of the low-frequency Discrete Fourier Transform (DFT) magnitudes. A DFT rewrites a series as a sum of waves. Keeping only the slow waves captures an event's overall lifecycle and discards day-to-day jitter. Dropping phase makes the comparison ignore reporting lags (Aw et al., Klementiev et al.). The second is the peak lagged correlation within a window of a few days. The third is a Dynamic Time Warping (DTW) distance, which stretches one series against the other to absorb uneven delays (Fung and McKeown). For entity pairs we add a relation-temporality feature, which correlates the time series of each entity's co-mentions with a shared third entity. Lee and Hwang showed this is far less noisy than raw mention counts, and it directly targets the coincidental-spike failure.
The fusion is deliberately small:
$$ p(\text{same}\mid x) = \sigma\big(a\,\ell_J(x) + b^\top \phi(x) + c\big), $$
Here $\ell_J(x)$ is the judge's log-odds that the pair matches, read from its token probabilities or a verbalized confidence. $\phi(x)$ is the vector of burst features. $\sigma$ is the logistic function, and $a$, $b$, $c$ are a handful of weights fit on high-resource pairs only. The judge itself is never trained. With so few parameters, a transfer failure to low-resource languages would show that the evidence does not carry over. It could not be blamed on overfitting.
The data come from GlobeSumm's 370 events with 4,687 articles in 26 languages. Positive pairs are cross-lingual articles about the same event. Negatives are cross-lingual articles about different events in the same time window. These hard negatives are exactly the coincidental-burst cases where classic burst correlation is known to fail. For Indian languages we add the PIB corpus. Its temporally windowed, translation-based alignment collapsed to 2% accuracy for Oriya, which makes it a natural test of whether burst evidence helps where translation does not. Entity-level pairs are mined from the same streams in the manner of temporally correlated burst mining, and we hand-label a test sample.
Baselines
The naive reference point is a judge-free system that predicts "same" whenever two items share named entities after transliteration and fall within a fixed date window. It shows how much of the task is trivial. Accuracy reported against hard negatives specifically tells us whether the benchmark is hard enough to matter.
The strongest existing competitor is a frontier LLM judge used alone, under the best of three prompting regimes. These are a direct native-language comparison, translate-then-judge in English, and GlobeSumm-style protocol-guided prompting, which breaks reports into key facts before comparing them (GlobeSumm). It is the strongest competitor because generative judges are the best-performing evaluators in multilingual preference tests: GPT-4 Turbo averages 83.5% accuracy across 23 languages on M-RewardBench. GPT-4 also reaches 96.8% conflict-resolution accuracy on GlobeSumm under protocol-guided prompting, but that is a different task. No reported result was found on Lacuna for cross-lingual same-event pair judgment on GlobeSumm. We will therefore establish this baseline ourselves. For cross-lingual consistency the external reference is a Fleiss' Kappa of about 0.3 on average and 0.54 at best (GPT-4o on WikiLingua, yes/no criterion) from How Reliable is Multilingual LLM-as-a-Judge?. The judge baseline gets the same pairs, the same judge model, and the same number of prompt-tuning trials on the high-resource development split as the fused system. Beating it means a higher area under the ROC curve on hard negatives in the low-resource tier, with confidence intervals that do not overlap, and no loss in the high-resource tier.
The second strong competitor is classic burst correlation alone, a logistic model over $\phi(x)$ without $\ell_J$. It is the strongest language-agnostic method. If fusion does not beat it, the judge adds nothing. It is fit on exactly the same high-resource training pairs.
Two ablations isolate the key ingredient. The first replaces $\phi(x)$ with publication-date difference and same-week indicators, which tests whether burst shape matters or only timing does. The second keeps the burst information but moves it into the prompt as a short textual summary of the correlation, lag and DTW statistics, plus a downsampled series. This tests whether late fusion matters or whether the judge can integrate the evidence itself.
Why not just…?
Why not just use burst correlation alone, as the classic work did? It is cheap and language-independent, and on synchronized high-volume streams it can rival transliteration models (Tao et al.). Its known weakness is coincidental co-bursts over long timelines. Our hard negatives are built from exactly those, so we expect it to fail there. The burst-only baseline will show directly whether it does. If burst-only already matches fusion on hard negatives, the judge is redundant and we will report that.
Why not just put the time series in the judge's prompt? This is the most serious alternative. TSQueryBench found that judges rank time-series explanations far better than they generate them (0.84 versus 0.38 accuracy for Qwen-3 8B), which suggests they may read such evidence well. Against that, the same paper reports a blind spot on volatility and hallucinated anomalies, and its series were synthetic and single-stream. The Silent Judge shows that judges shift verdicts on cues while acknowledging them 0% of the time, so prompt-injected evidence may be over-trusted or ignored without any visible trace. The prompt-injection ablation, together with the counterfactual swap described below, settles this. If in-prompt evidence matches late fusion and the judge responds to swapped evidence in the right direction and magnitude, the simpler design wins and the claim narrows to "burst evidence helps judges."
Why not just use publication dates? Same-week publication is a crude temporal signal that a reviewer could reasonably expect to capture most of the benefit. Hard negatives share a time window by construction, so dates alone should not separate them. The date-only ablation tests this. If it comes within noise of the burst features, burst shape is not doing the work and the proposal fails.
Why not just fix the judge internally, as DIBJudge does? DIBJudge reduces translationese bias by about 80% on BELEBELE, but it needs access to model internals and retraining. It targets a bias about style, not whether two texts describe the same real-world referent. Our method works on closed judges and is complementary to it. We will run DIBJudge-style debiasing as an additional judge variant where open weights allow, and check whether the burst gain survives on top of it.
Proposed Experiments
The main experiment reports area under the ROC curve and accuracy on all pairs and on hard negatives, broken out by language resource tier. It also reports expected calibration error, because a fused probability is only useful if it is honest. We run it with two judges, one closed frontier model and one open model such as Qwen-2.5-72B, to show the effect does not depend on a single model.
The cross-lingual consistency experiment follows the protocol of How Reliable is Multilingual LLM-as-a-Judge?. For each GlobeSumm event we pose the same correspondence question against an English anchor once per language. We then compute Fleiss' Kappa across languages for the judge alone and for the fused verdict. The prediction is a clear rise from the roughly 0.3 regime, concentrated in low-resource languages.
The decisive check is a counterfactual evidence swap in the prompt-injection condition. We replace a pair's burst summary with one taken from a mismatched pair and measure the verdict shift rate, as in The Silent Judge. We also record whether the judge's explanation mentions the evidence. Near-zero shift means the judge ignores the evidence. Large shift with no acknowledgment means it is a silent shortcut. Either outcome supports late fusion over prompting.
A second check probes the main threat to the core intuition. Low-resource languages may have both weaker judges and sparser, noisier streams, in which case the two witnesses fail together rather than separately. We will measure the correlation between judge errors and burst errors per tier, and plot fused gain against monthly stream volume. If errors are strongly correlated where volume is low, the complementarity argument breaks. We would then report the volume threshold below which burst evidence stops helping, which is itself a useful practical result. The selective temporality idea of restricting comparison to informative windows is the planned fallback in that regime.
References
- How Reliable is Multilingual LLM-as-a-Judge?
- Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck
- Named Entity Transliteration and Discovery from Multilingual Comparable Corpora
- Unsupervised Named Entity Transliteration Using Temporal and Phonetic Correlation
- TSQueryBench: LLM-as-a-Judge for Time-Series Explanations
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization
- Feature-Based Method for Document Alignment in Comparable News Corpora
- Aligning Noisy Parallel Corpora Across Language Groups: Word Pair Feature Matching by Dynamic Time Warping
- Understanding Relation Temporality of Entities
- A Multilingual Parallel Corpora Collection Effort for Indian Languages
- Mining named entities with temporally correlated bursts from multilingual web news streams
- M-RewardBench: Evaluating Reward Models in Multilingual Settings
- Enriching Entity Translation Discovery using Selective Temporality