Entity Centroids as the Unit of Comparison in Coreference

Coreference resolution means finding which phrases in a document refer to the same real-world thing. Today's neural systems pay a quadratic price to do this. They score every candidate phrase (a "span") against every earlier span to pick its antecedent, the earlier phrase it refers back to. To keep this affordable they discard most candidates early, and the discarded true mentions can never be recovered. We propose a different unit of comparison. During training, the span embeddings of each entity are pulled toward that entity's centroid, the mean of its mention embeddings computed on the fly inside the document. At inference, each new span is compared only to the centroids of the entities found so far, not to every earlier span. The cost then grows with the number of entities rather than the number of mentions. That saving can be spent on discarding fewer candidates in the first place.

The idea joins three threads. The first is center-style losses that make same-class embeddings compact, previously used only with a fixed set of classes. The second is the cost of comparing all pairs of spans. The third is the fragility of combining several loss terms with hand-set weights. The weighting problem is not treated as decoration. Adding a third loss to a coreference model is exactly the kind of change that makes training sensitive to scalar weights, so the proposal measures that sensitivity directly. It also tests whether a scale-free form of the compactness loss removes the need to tune.

Core Claim

A coreference model trained with a document-local, ratio-form compactness loss can replace pairwise antecedent scoring with mention-to-entity-centroid comparison. Doing so costs no accuracy on OntoNotes. Because the comparison is cheaper, the model can afford to keep roughly twice as many candidate spans at the same compute budget. Keeping more spans should raise gold-mention recall above the "over 92%" ceiling reported for top-40% pruning by End-to-end Neural Coreference Resolution. We expect that recall gain to turn into F1.

The new elements are three. Class centers are defined per document from the clusters themselves, so no stored prototype table is needed. Compactness is enforced by a loss rather than by architecture. And the compact geometry is used to change the inference-time comparison structure. Prior work has done none of these together.

Relation to Prior Work

Three papers motivate the proposal by showing the gap.

End-to-end Neural Coreference Resolution introduced the now-standard design. It scores each span on its own, keeps the top 0.4N spans, and scores pairs among the survivors. The authors observe false links between semantically similar but distinct entities, such as "pilots" and "flight attendants." That observation suggests mention embeddings of one entity are not well separated from those of a neighbouring entity.

Coreference Resolution with Entity Equalization is the closest work. It replaces each mention representation with a soft-cluster aggregate and reports a gain from 73.0 to 76.61 average F1. That gain is evidence that entity-level pooling helps. Entity Equalization, however, changes the architecture and keeps pairwise antecedent scoring at test time. We instead shape the geometry with a loss and then drop pairwise scoring at inference.

Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering adds a mention-detection loss with a fixed weight. It reports 67.8 F1 for a single model and a 0.3-point drop when that loss is removed. This shows that the extra loss terms coreference relies on are small, weight-dependent effects.

Two papers supply the method.

Contrastive-center loss for deep neural networks defines a ratio loss. Its numerator is a sample's squared distance to its own class center, and its denominator is the sum of squared distances to the other centers. The paper reports center separations 3.3 to 5 times larger than plain center loss. It assumes a fixed class set with learned centers and names the memory cost of large class counts as a limitation. We reuse the ratio form but compute the centers as in-document means, which removes the class table entirely.

Multi-Task Learning as Multi-Objective Optimization gives MGDA-UB. This method combines task gradients at the shared representation into a minimum-norm direction in a single backward pass. It reports 8.25% average CelebA error against 9.62% for uniform weighting. We use it as a competitor for loss balancing, not as our method.

Two papers act as constraints.

Domain Separation Networks and Generating Images with Perceptual Similarity Metrics based on Deep Networks both report that equilibria among several losses were fragile and needed dataset-specific weights. Any new auxiliary term must therefore be shown not to need careful tuning.

SpanBERT sets the encoder and the accuracy bar. It still uses the pruned pairwise design.

Research Program

The model starts from the SpanBERT coreference pipeline in the facebookresearch/SpanBERT codebase. It keeps that pipeline's span representations, mention scores and antecedent loss. During training, for each document with gold clusters $k = 1..K$, we compute the centroid $c_k$ as the mean embedding of the retained gold-mention spans in cluster $k$. For a retained span $g_i$ belonging to cluster $y_i$, the compactness term is

$$ \mathcal{L}_{\text{cc}} = \frac{1}{|S|}\sum_{i \in S} \frac{\|g_i - c_{y_i}\|^2}{\sum_{k \neq y_i}\|g_i - c_k\|^2 + \delta}, $$

where $S$ is the set of retained gold mentions in non-singleton clusters and $\delta$ is a small constant. Stop-gradient is applied to $c_{y_i}$ when it is computed from fewer than two other members, so that a two-mention cluster cannot collapse trivially.

The total loss adds $\lambda\,\mathcal{L}_{\text{cc}}$ to the antecedent likelihood. The ratio is dimensionless and bounded near the range of a probability. Our hypothesis is that its gradient scale stays comparable to the likelihood term across documents of very different lengths, whereas plain center loss grows with embedding norm. This is the concrete sense in which the proposal addresses weight sensitivity.

Inference proceeds left to right over the retained spans. The model keeps a running mean embedding for each entity found so far. A span joins the nearest entity if a learned score clears a threshold. The score combines distance to the centroid with the span's mention score, and the threshold is fit on the development set. Otherwise the span starts a new entity. Each span therefore costs one comparison per existing entity. On OntoNotes, entities are typically far fewer than mentions, so total comparisons scale as retained spans times entities, $O(MK)$, against $O(M^2)$ for antecedent scoring over $M$ retained spans. We will measure $K/M$ on the development set rather than assume it.

The analysis has three parts. First, we measure geometry: intra-entity variance and nearest-other-entity distance before and after training. Second, we measure sensitivity: development F1 as a function of $\lambda$ over four orders of magnitude, for the ratio loss and for plain center loss. Third, we measure the recall–compute trade-off: gold-mention recall and F1 as the kept fraction rises from 0.4 to 0.8 under a fixed budget of comparisons.

Baselines

The naive reference point is the unmodified SpanBERT coreference model from SpanBERT, retrained in our environment with its released settings. That paper reports 79.6 average F1 on the OntoNotes / CoNLL-2012 benchmark. It does not specify model size in the material available to us, so we will reproduce it at the base and large sizes and compare against our own reproduction. This is also the strongest existing method, because it uses the same encoder and the same pruned pairwise design we modify. Beating it means matching its F1 within run-to-run variance while using fewer pairwise comparisons, or exceeding it at an equal comparison budget. We will use three or more seeds per configuration, with significance measured over seeds.

The closest architectural competitor is Entity Equalization from Coreference Resolution with Entity Equalization, which reports 76.61 F1 with a BERT encoder. We will reimplement it on the SpanBERT encoder so that the comparison isolates pooling-by-architecture against compactness-by-loss. It gets the same training steps and the same development-set tuning budget, which is eight configurations per method.

Two ablations isolate the key ingredient. The first applies centroid-based inference to the unmodified SpanBERT model. If that already matches pairwise scoring, the loss is unnecessary and the claim fails. The second trains with the compactness loss but keeps pairwise inference. This shows whether any gain comes from better embeddings or from the cheaper comparison that allows looser pruning.

For loss balancing, we compare fixed $\lambda$ with a five-point grid against MGDA-UB from Multi-Task Learning as Multi-Objective Optimization, applied at the span-embedding layer. MGDA-UB gets no extra tuning. The ratio loss counts as better on robustness if its development F1 varies by less than half a point across the full $\lambda$ sweep while plain center loss does not.

Why not just…?

Why not just use the stronger pretrained encoder and leave the loss alone? Pretraining alone does not settle entity-level grouping. Coreference Resolution with Entity Equalization gained 3.6 points by adding entity pooling on top of BERT. The first ablation above tests this directly: if centroid inference on unmodified SpanBERT embeddings loses no F1, this alternative wins and we will say so.

Why not just add an in-document contrastive or InfoNCE loss over mention pairs? This is a real threat. Within one document, the retained spans number only a few hundred, so a pairwise contrastive loss is affordable. Unlike the image setting analysed in Contrastive-center loss for deep neural networks, the cost argument against pairs is weak here. Such a loss may produce equally compact clusters. We will train this variant with an equal tuning budget and run the same centroid inference on it. If it matches our loss, the contribution narrows to the claim that compact geometry permits centroid inference, whichever loss produces it. We would report it that way rather than claim the ratio loss is special.

Why not just grid-search $\lambda$? For a single added weight, a five-point grid is cheap. If it reliably finds a good value, the weight-sensitivity concern is minor. Exhaustive search only becomes impractical when many objectives are combined, as in the 40-task setting of Multi-Task Learning as Multi-Objective Optimization. We include the grid as a baseline. The balancing claim is deliberately modest: the ratio form should make the grid unnecessary and transfer across encoder sizes without re-tuning. We will test this by moving the base-model $\lambda$ to the large model unchanged.

Why not just keep more spans in the pairwise model? Keeping a fraction 0.8 instead of 0.4 roughly quadruples pairwise comparisons. We run this configuration explicitly. If its F1 gain matches ours, the comparison budget, not the geometry, is what matters. That result would still quantify how much the recall cap of End-to-end Neural Coreference Resolution costs.

Proposed Experiments

The primary benchmark is OntoNotes / CoNLL-2012, scored as average F1 over the MUC, B³ and CEAF scores. We also report gold-mention recall after pruning and the number of span-to-span or span-to-centroid comparisons per document. The reference numbers from the literature are 79.6 F1 for SpanBERT, 76.61 for Entity Equalization, 67.8 for the biaffine model, and 67.2 for the original end-to-end model.

The decisive check is a plot of F1 against comparisons per document for four systems. The first is pairwise SpanBERT at kept fractions 0.4, 0.6 and 0.8. The second is centroid inference without the compactness loss. The third is centroid inference with the pairwise contrastive loss. The fourth is centroid inference with the ratio loss. The proposal succeeds if the ratio-loss curve lies above the pairwise curve, meaning higher F1 at equal comparisons or equal F1 at fewer comparisons, and above the no-loss ablation. It fails if centroid inference without the loss already matches, or if the gap disappears once pairwise SpanBERT is given the same comparison budget.

We will break down the analysis by document length, where savings over $O(M^2)$ should be largest, and by mention length above five tokens, the category Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering found most often lost to pruning.

The second decisive check is the $\lambda$ sensitivity sweep for ratio loss against plain center loss, together with the MGDA-UB comparison.

As a transfer probe, the same document-local centroid loss can be applied to record linkage on the entity-matching benchmarks (Abt-Buy, Amazon-Google, DBLP-ACM) used by Cost-Efficient RAG for Entity Matching with LLMs, using blocks as the "documents." We found no reported result on Lacuna for a centroid-trained matcher on these datasets, so this probe is exploratory and secondary to the coreference results.