Background-Subtracted Shower Moments for Cross-Lingual Article Similarity

Deciding whether two news articles in different languages report the same story is usually done by turning each article into a vector and taking a dot product, or by feeding both into one transformer and regressing a score. Both approaches fail in two documented ways. First, accuracy swings widely from one language pair to the next: on the SemEval-2022 multilingual news benchmark, median system correlation was 0.82 on French–Polish but 0.60 on German–French. Second, models are fooled by pairs that share politicians and places but describe different events. Gamma-ray astronomers face a structurally similar problem. A ground telescope photographs faint flashes of Cherenkov light, the blue glow given off when a particle cascade from a cosmic gamma ray outruns light in air. Each real flash must be separated from about ten thousand times as many flashes caused by ordinary cosmic rays. The standard pipeline first subtracts the night-sky glow measured for the current observing conditions. It then removes isolated pixels and describes what remains with a few shape statistics: total brightness, length, width, and orientation relative to the source.

We propose applying that pipeline to the sentence-by-sentence similarity grid of an article pair. Two articles telling the same story should produce a bright, elongated streak that runs roughly along the diagonal of the grid, because matching events appear in matching order. Articles that only share entities should produce scattered, diffuse spots. The step most relevant to cross-lingual variation is the background subtraction. The baseline level and spread of sentence similarities for a given language pair can be estimated from unrelated article pairs, which requires no labels. Subtracting it should therefore correct pair-specific offsets even for language pairs never seen in training.

Core Claim

Much of the per-language-pair variation in cross-lingual document similarity comes from pair-dependent offsets in the similarity values themselves, rather than from missing semantic knowledge. Embedding similarities between, say, German and Polish sentences sit at a different baseline and spread than those between French and Spanish. A model that learns fixed thresholds on eight training pairs therefore miscalibrates on the ten unseen test pairs. We claim three things. Subtracting a per-pair background estimated from unlabeled text, before any shape analysis, should reduce this variation. A handful of shape statistics computed on the cleaned grid should separate true narrative matches from entity-overlap matches better than a pooled vector does. Finally, adding these few interpretable features to a strong encoder should raise worst-pair and unseen-pair correlation without lowering overall correlation.

The closest existing work is WueDevils at SemEval-2022 Task 8. It builds the same sentence-similarity grids from Universal Sentence Encoder and Sentence-BERT embeddings and feeds them raw into a convolutional network plus a random forest, reaching 0.759 Pearson. That system does no per-pair background correction and no explicit shape parameterization. It also reports a drop to 0.64 on Chinese–Chinese, which it attributes to sentence-splitting errors. Our proposal differs in exactly the two steps the astronomy pipeline treats as essential.

Research Program

The image-processing steps come directly from the telescope literature. The pixel cleaning and moment analysis are those of the cta-lstchain pipeline validated on the Large-Sized Telescope prototype (Observations of the Crab Nebula and Pulsar with the LST prototype). The moments are the classical Hillas parameters (Upper Limit for γ-Ray Emission from Draco).

For an article pair in languages $(a,b)$, let $S_{ij}$ be the cosine similarity between sentence $i$ of the first article and sentence $j$ of the second, taken from a frozen multilingual sentence encoder. Let $\mu_{ab}$ and $\sigma_{ab}$ be the mean and standard deviation of sentence similarities over randomly re-paired, unrelated articles in the same two languages. The background-subtracted grid is

$$Z_{ij} = \frac{S_{ij} - \mu_{ab}}{\sigma_{ab}}.$$

We then apply two-level cleaning. A cell survives if $Z_{ij}$ exceeds a high threshold, or if it exceeds a lower threshold and touches a surviving neighbor. Sentence indices are rescaled to $[0,1]$ so that articles of different lengths share coordinates. On the surviving cells, weighted by $Z_{ij}$, we compute:

  • size: total surviving intensity;
  • centroid: where in the two articles the match lives;
  • length and width: square roots of the two eigenvalues of the intensity-weighted covariance;
  • alpha: the angle between the major axis and the main diagonal;
  • concentration: the fraction of intensity in the brightest cells.

These features yield testable predictions. A same-story pair should have large size, a high length-to-width ratio, and small alpha. An entity-overlap pair should have moderate size spread over a round, wide blob.

The centroid also offers a direct, per-pair measurement of a tension in the literature. HFL and the task overview find that titles and lead paragraphs carry the signal. Nikkei finds that truncating the article body hurts. A centroid near the start of both articles would favor the first view, and one deep in the bodies would favor the second.

The resulting roughly ten features are concatenated with the score of a strong fine-tuned XLM-RoBERTa model and passed to a small gradient-boosted regressor. This regressor learns a correction to the encoder score rather than replacing it.

The second strand of the program is diagnostic. For each of the 18 test language pairs we record three things:

  1. the per-pair correlation before and after background subtraction;
  2. typological and resource distances from DistaLs, which found correlations of −0.42 to −0.45 between such distances and parsing accuracy;
  3. the tokenizer's average subword pieces per word for each language.

If our claim holds, background subtraction should shrink the share of per-pair variance that these distances explain. Eighteen pairs is a small sample, so we will report bootstrap intervals and treat this analysis as supporting evidence, not proof.

Baselines

The naive reference point is the cosine similarity between mean-pooled multilingual article embeddings, with no fine-tuning and no calibration. It shows how much any learning adds at all.

The strongest existing method is the first-place system, HFL. It is an XLM-RoBERTa regressor trained with translate-train and back-translation augmentation, a dropout-consistency regularizer using mean squared error, multi-dimension labels, and head-plus-tail inputs. It reports 0.818 Pearson correlation on the official test set and 64.31 on German-to-Polish. We will reimplement it on the official training split, confirm we are within about 0.01 of 0.818, and use our reproduction as the encoder inside our system. Our gain then measures only the added features. The tuning budget for the correction regressor will be matched by giving the HFL reproduction an equal number of extra hyperparameter trials.

We also include two other published systems. The bi-encoder of Nikkei reports 0.7107 for a single model and 0.7425 for an ensemble. It matters here because Nikkei, unlike HFL, found that bi-encoders beat cross-encoders (0.6188), and calibration effects may differ between the two. The SimCNN system of WueDevils, at 0.759, is the direct comparison for learned versus parameterized grid analysis.

Three ablations isolate the key ingredient:

  • moments without subtraction: shape statistics computed on the raw grid;
  • subtraction without shape: only the background-subtracted mean similarity is added to the encoder score;
  • CNN on the cleaned grid: the SimCNN architecture applied to the background-subtracted, cleaned grid.

Beating the field means meeting both of two criteria with paired bootstrap significance. Overall Pearson must be no lower than our HFL reproduction. Correlation on the ten language pairs absent from training, along with the worst-pair correlation, must improve by at least 0.03.

Why not just…?

Why not just calibrate the final score per language pair? This is the most serious threat. Z-scoring the encoder's output score within each language pair, using statistics from unlabeled re-paired articles, is far simpler and also needs no labels. Our hypothesis is that a pair-dependent offset distorts more than the final scalar. It changes which sentence cells survive cleaning, and therefore the measured shape. We will run output-level calibration as its own baseline. If it recovers most of the unseen-pair gain, we will report that the contribution is calibration alone and drop the shape claims.

Why not just run a CNN on the grid? Astronomy itself warns against hand-made moments. Convolution and Graph-based Deep Learning Approaches for Gamma/Hadron Separation shows that hybrid CNN-graph models beat Hillas-only classifiers on simulated H.E.S.S. data, with a further 9% false-positive reduction from adding shower-core information. Those models, however, are trained and tested on the same simulated distribution. That paper explicitly flags shift to real instruments as unquantified. Our test set, with ten unseen language pairs, is exactly such a shift. The CNN-on-cleaned-grid ablation settles the question. If it matches the moments on unseen pairs, we keep the CNN and credit only the cleaning step.

Why not just fine-tune a cross-encoder harder, or translate everything to English? HFL already squeezes augmentation hard and still collapses to 64.31 on German-to-Polish. Nikkei found that translating everything to English gave no improvement over native multilingual encoding. Our system sits on top of the HFL model, so any gain is additive to that approach.

Why not prompt a large language model? Few-shot demonstrations in non-English languages often fail to beat zero-shot prompting (The Impact of Demonstrations on Multilingual In-Context Learning). We found no reported result on Lacuna for LLM prompting on SemEval-2022 Task 8. We will therefore include a zero-shot prompted LLM as a reference baseline and not assume it is weak.

Proposed Experiments

All experiments use the SemEval-2022 Task 8 data (task overview), with 4,964 training pairs over 8 language pairs and 4,593 test pairs over 18. The metrics are overall Pearson correlation, per-pair Pearson, Pearson on the narrative sub-score, and the variance of per-pair correlations.

The first decisive check targets entity overlap. We select test pairs whose entity sub-score is high but whose narrative sub-score is low, using the benchmark's sub-dimension labels. On this subset we ask whether the length-to-width ratio and alpha separate those pairs from true narrative matches better than the HFL score alone. Separation is measured by AUC (area under the ROC curve).

The second check targets segmentation brittleness, in the spirit of the script-level stress tests of NOLLI. We randomly merge and split sentences in the Chinese and other non-Latin-script articles and measure how much each system's score changes. Moments on rescaled coordinates should change less than SimCNN on raw grids.

The third check is the DistaLs variance analysis described in the Research Program.

The proposal fails cleanly in either of two cases. Output-level calibration may match the full system on unseen pairs. Alternatively, the shape features may add nothing on the entity-overlap subset beyond what subtraction alone provides. Either outcome is itself a useful, publishable answer about where the language-pair gap comes from.