Relations as Anomalies: Where the Signal Hides in Entity-Pair Representations

Document-level relation extraction asks a model to read a document, consider every pair of entities it mentions, and say which pairs are related and how. Most of that work is saying "no". In DocRED more than 97% of candidate pairs carry no relation (DocRED). When a relation is known to exist, current models name its type at roughly 90+ F1 (Revisiting DocRED). The hard part is deciding whether any relation exists. This proposal treats that decision as anomaly detection. The unrelated pairs are the "normal" population, and a true relation is a departure from it.

We borrow one finding from the feature-redundancy literature. In deep features, the directions with the most variance mostly describe the normal population. The directions that separate rare cases are few and scattered across the spectrum (Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection). Our claim, in two sentences: if you fit a Gaussian to the vectors of unrelated entity pairs, the evidence that a pair is related sits in a handful of low-variance directions of that Gaussian. Picking those few directions gives a relation-existence detector that needs very little labeled data, and it can find the true relations that incomplete annotation mislabeled as "no relation".

Core Claim

Consider the pair vector that a strong document-level model such as ATLOP builds just before its classifier. We estimate the mean $\mu$ and covariance $\Sigma$ of these vectors over pairs labeled "no relation" (NA), and eigendecompose $\Sigma$ into directions $u_j$ with variances $\lambda_j$. We score a pair vector $z$ with a Mahalanobis distance restricted to a small selected set $S$ of directions:

$$s_S(z) = \sum_{j \in S} \frac{\left(u_j^\top (z-\mu)\right)^2}{\lambda_j}$$

A high score means the pair looks unlike the unrelated-pair background. We make three predictions. First, the set $S$ that best separates related from unrelated pairs is small, on the order of tens of directions out of about a thousand. Second, $S$ lies mostly outside the top principal components, so ordinary PCA would discard it. Third, $s_S$ is more robust than a discriminatively trained existence classifier in two regimes: when labels are scarce, and when the NA labels themselves contain hidden positives.

The second regime matters because DocRED's annotation omitted about 64.6% of true triples (Revisiting DocRED). A classifier trained on those labels is explicitly taught to call missing relations "NA". A background density fitted with a robust, trimmed covariance absorbs a small contamination rate without being directly told that those pairs are negatives.

How This Differs From Prior Work

The anomaly-detection recipe comes from Gula et al. (Greedy Eigencomponent Selection). They fit a Gaussian to normal image features and greedily choose eigencomponents by AUROC rather than by eigenvalue. They reached near-perfect detection with fewer than 10 components on MVTec-AD, and adding high-variance components hurt. We reuse their selection procedure and inherit its main limitation: selection needs some labels. That limitation fits our setting, because document-level extraction always comes with at least a few annotated documents. Nobody in the file has applied this recipe to text, to entity pairs, or to label noise.

The closest relation-extraction work is Semantic Consistency (Han & Sun). It decomposes each distantly supervised instance over subspaces spanned by the positive instances of each relation, using sparse hand-crafted features and $\ell_1$ minimization, and it removes false positives. We model the negative background with dense pretrained representations, and we aim at the opposite error: false negatives in document-level data.

MMIB (Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneck) compresses text and image features with an information bottleneck, a training objective that discards input information not needed to predict the label. It reached 83.23 F1 on multimodal relation extraction. It is trained end-to-end on sentence-image pairs and does not separate existence from type or deal with the NA imbalance. Our method is a post-hoc probe of a trained encoder's geometry. It is not a new training objective.

Two papers address the NA problem through the classifier head. ATLOP (Adaptive Thresholding and Localized Context Pooling) learns a per-pair threshold class. PRiSM (Relation-Aware Score Calibration) calibrates logits against embedded relation descriptions. Neither looks at where in representation space existence is encoded. We treat both as competitors and as heads that our score can be combined with.

We considered and rejected two other options from the file. A repulsive DPP prior on attention heads (Priors for Diversity in Generative Latent Variable Models) is too easily matched by simpler head-diversity penalties. A variational bottleneck on pair vectors is too close to MMIB.

Research Program

We work with three encoders: ATLOP with BERT-base, ATLOP with RoBERTa-large, and a frozen pretrained encoder with ATLOP's localized context pooling but no fine-tuning. The frozen encoder tests whether the effect comes from pretraining or from task training.

Step 1: spectral anatomy. For each encoder we extract pair vectors on the training data and fit the NA Gaussian. We compare two covariance estimators: a plain one, and a trimmed one that iteratively drops the top 3% of highest-scoring NA pairs. The 3% figure matches the approximate rate of hidden positives implied by the Re-DocRED statistics.

We then compute the AUROC of each single eigendirection, meaning its standardized squared projection, for separating related from unrelated pairs. We plot this per-direction AUROC against the direction's rank in the eigenspectrum. We run greedy forward selection with a budget of 64 directions. At each step we evaluate every remaining direction, which is cheap once projections are cached.

We also check whether $S$ is shared across relation types or splits by type. We do this by rerunning selection separately for frequent and long-tail relations.

Step 2: use the score for extraction. We factor prediction into two decisions. The existence decision uses $s_S$ passed through a monotone one-parameter calibration fitted on training folds. The type decision reuses the existing model's relation logits, restricted to pairs judged to exist. We also test a softer variant that adds $s_S$ as an extra input to ATLOP's threshold class.

Step 3: label-noise audit. We train encoders on original DocRED only. We then score every training pair that DocRED labels NA, and measure how well $s_S$ ranks the pairs that Re-DocRED later recovered as true triples.

Baselines

The naive reference point is a BERT-base ATLOP-style model with a single global threshold. In the low-resource regime, PRiSM reports 10.44 F1 for BERT-base on 3% of DocRED (PRiSM), and we will reproduce this number from the brightjade/PRiSM code before comparing.

The strongest full-data competitor is ATLOP with adaptive thresholding. It reports 63.4 F1 on DocRED (ATLOP). This is the right target because its threshold class already acts as a learned per-pair existence detector. Beating it with a probe on its own features would show that its learned threshold does not use the geometry well. Revisiting DocRED reports that ATLOP and three other architectures reach the mid-70s F1 on Re-DocRED, and we retrain ATLOP ourselves there. On CDR, the file gives CFER's Inter-F1 of 57.8% (Coarse-to-Fine Entity Representations) and says ATLOP improves on prior methods by 4.3 F1, with no absolute ATLOP number, so we reproduce ATLOP on CDR.

The strongest low-resource competitor is PRiSM. With BERT-base on 3% of DocRED it reports 37.02 F1 and an expected calibration error of 0.010%. We run it from the released code under the identical 3% and 10% splits, with the same seeds.

Fairness rules apply to every comparison. All methods use the same encoder checkpoints, the same training documents, and the same hyperparameter search budget of 20 configurations. Our selection of $S$ uses only training labels, via cross-validation folds. The development set is used only for the shared search, never for selecting directions.

We need three ablations to isolate the key ingredient, which is choosing directions by task relevance rather than by variance:

  1. The full Mahalanobis distance over all directions.
  2. Mahalanobis restricted to the top-$k$ principal components, with $k=|S|$.
  3. A random set of $k$ directions.

Two further ablations fit the Gaussian on all pairs instead of NA pairs only, and swap trimmed for plain covariance.

What counts as beating the baselines depends on the regime. In the low-resource regime, the method must exceed PRiSM's F1 on the same split by more than the seed-to-seed standard deviation, without worse calibration error. In full data, the realistic bar is matching ATLOP's F1 when combined with its type head. On the noise audit, the method must beat ATLOP's own NA-versus-rest margin at ranking the recovered triples. No reported result was found on Lacuna for false-negative detection AUROC on DocRED, so that comparison is established here.

Why not just…?

Why not just use PCA or whitening on the pair vectors? Gula et al. found that top-variance components of deep features describe the normal population and that PCA truncation degrades detection. That finding is from images, not text, so we do not assume it transfers. The top-$k$ PCA ablation and the per-direction AUROC-versus-rank plot test it directly. If the leading components turn out to be the discriminative ones for text pair vectors, the selection step adds nothing and we will report that.

Why not just train a class-weighted logistic "relation exists" classifier on the pair vectors? This is the most serious threat. With full, clean labels, as in Re-DocRED, it should plausibly match our score, and we do not claim otherwise. Our claim is limited to two regimes. In low resource, selection chooses tens of discrete directions instead of fitting about a thousand weights, which gives far fewer degrees of freedom. In the noisy-NA regime, the discriminative loss directly pushes hidden positives toward NA. We run this classifier, with and without an L2 penalty tuned on the same budget, in every setting. If it matches our score both at 3% data and on the DocRED false-negative audit, the anomaly framing has no practical value and the contribution shrinks to the spectral analysis.

Why not just rely on ATLOP's adaptive threshold or PRiSM's calibration? Both adjust scores at the output. PRiSM notes that temperature scaling fails because one scale factor fits all classes. Neither method changes what information the existence decision can draw on. The combined variants test whether our score is complementary, meaning ATLOP's threshold plus $s_S$ outperforms either alone. If adding $s_S$ gives nothing on top of PRiSM, the redundancy view is subsumed by calibration.

Decisive Checks

We run everything on DocRED at full data and at 3% and 10% training splits, on Re-DocRED, and on CDR. CDR tests transfer to a biomedical domain where the background population is different.

Check 1: where the signal lies. The first check is whether greedy selection reaches at least the full-Mahalanobis AUROC for relation existence with at most 32 directions. We also check whether most selected directions fall outside the top 32 principal components. If both fail, the core claim is false.

Check 2: false-negative recovery. Among DocRED-labeled NA pairs, do pairs that Re-DocRED marks as true triples score higher under $s_S$ than under ATLOP's margin and the logistic existence classifier? We measure this with AUROC and precision at the top 1% of pairs. We also report how precision changes as the trim rate varies.

Check 3: low-resource gains. At 3% DocRED, does the factored model, with the selected-direction existence score plus the type head, beat PRiSM's reproduced F1? We report expected calibration error alongside F1. We also break results down into intra-sentence and inter-sentence pairs. This shows whether scattered low-variance directions carry multi-hop evidence or only local cues.

A clear positive result would give document-level extraction a cheap, label-efficient existence detector and a tool for auditing its datasets. A negative result, particularly if the logistic probe matches us everywhere, would still establish that dense pair vectors do not share the variance-utility mismatch seen in image features. That would be useful for anyone tempted to compress them with PCA.