Metrics such as BLEU and ROUGE score generated text by counting shared words and short word sequences with a reference. They fail in two ways. They penalize correct paraphrases that use different words, and they reward word salads that reuse the reference's words. Dense sentence embeddings were meant to fix this, but they still treat "how far apart two sentences are" as a single global distance. In that distance, rewording and meaning change are mixed together.
This note proposes to separate the two directly, and to do it locally. Around any region of meaning space, some directions of movement in a frozen encoder's embedding space correspond to harmless rewording. Other directions correspond to real changes in meaning. Borrowing from differential geometry, we call the rewording directions the fiber: the set of surface forms that all express one meaning. The idea is to learn a small atlas of charts, meaning overlapping local regions of embedding space, each with its own estimate of the rewording directions. The charts are blended smoothly where they overlap. Two sentences are then judged equivalent by how far apart they are after the local rewording directions are projected out.
The central empirical bet is simple and falsifiable. It is that rewording directions differ across regions of meaning. Paraphrasing a question about numbers should involve different directions than paraphrasing a biographical sentence. If that is false, one global projection is enough, and the atlas adds nothing.
Core Claim
Take a frozen sentence encoder $e(\cdot)$ mapping text to $\mathbb{R}^d$. Let $K$ chart centers $c_k$ define soft responsibilities $\psi_k(x)=\mathrm{softmax}_k(-\|x-c_k\|^2/\tau)$ with temperature $\tau$. These responsibilities form a partition of unity: weights that sum to one and let neighboring charts share responsibility where they overlap. Each chart carries a rank-$r$ projector $P_k$ onto its local rewording (fiber) subspace. The proposed equivalence distance between sentences $a$ and $b$, with midpoint $m=(e(a)+e(b))/2$, is
$$ d_{\text{fib}}(a,b)=\sum_{k=1}^{K}\psi_k(m)\,\big\|(I-P_k)\,(e(a)-e(b))\big\|_2 . $$
The claim is that $d_{\text{fib}}$ with $K>1$ separates paraphrases from high-overlap non-paraphrases better than three alternatives. The first is plain cosine distance. The second is a single global projector ($K=1$). The third is hard, non-overlapping clusters. We further claim that it approaches an end-to-end fine-tuned encoder while training only a few thousand parameters per chart and transferring better across domains.
The same distance plugs directly into content-based diversity metrics. There, it should stop counting paraphrased duplicates as diversity, while still rejecting shuffled word salads.
Relation to Prior Work
The atlas machinery comes from generative modeling on non-Euclidean data. Learning a Manifold as an Atlas represents a manifold as overlapping affine patches to avoid tearing. Atlas Generative Models and Geodesic Interpolation glues chart-specific generators with a partition of unity on MNIST. Implications of data topology for deep generative models shows, using persistent homology, that chart autoencoders recover structure that a single Euclidean latent space cannot. We reuse the partition-of-unity gluing and the "locally flat, globally not" assumption. We apply them to a question these papers never ask: whether the invariance structure of meaning is itself local.
The closest conceptual match is Bundle Networks. It models a many-to-one map as a fiber bundle, uses k-means neighborhoods as local trivializations, and trains an invertible network. It was tested only on tori, Möbius bands and tabular regression, and its neighborhood KL loss costs $\mathcal{O}(n^2)$ in neighborhood size. We deliberately drop its generative and invertible components, because evaluation does not need to sample the fiber, only to project it out. This is what makes the idea cheap enough to run on text.
On the text side, Unsupervised Sentence-embeddings by Manifold Approximation and Projection (EMAP) applies global UMAP to Word Mover's Distance matrices. It uses a single chart built on a bag-of-words transport distance. Because that distance ignores word order, scrambled sentences with identical words sit at zero distance. We differ on both counts: our charts are local, and we build on a contextual encoder.
PAWS supplies both the motivation and the supervision. It shows that BERT falls from over 90% on QQP to 33–40% on word-swapped pairs, and recovers to 85% after training on 12,000 such pairs. Its negatives are exactly the displacements that must not lie in any fiber.
BERTScore and BLEURT are the learned-metric competitors. BERTScore uses greedy token cosine matching. BLEURT is a cross-encoder pre-trained on 6.5M perturbed pairs and then fine-tuned on human ratings. Neither separates rewording from meaning change explicitly. Evaluating the Evaluation of Diversity in Commonsense Generation provides the diversity testbed on which content-based embedding metrics beat form-based ones.
Research Program
Measure the structure. The first step is a diagnostic that can kill the project cheaply. We embed QQP and PAWS-Wiki paraphrase pairs with frozen encoders: BERT, and the SimCSE model used in the diversity study. We partition the embeddings with k-means and compute, in each cluster, the top-$r$ principal directions of the paraphrase difference vectors $e(a)-e(b)$. We then measure principal angles between clusters' subspaces, and compare them against a null distribution. The null comes from randomly reassigning pairs to clusters of equal size.
If the cluster subspaces are no more different than the null, rewording is globally structured. In that case we report this as the finding and stop the atlas program. If they differ substantially, the atlas has something to model.
Learn the charts. Each $P_k$ is initialized from the local principal directions. It is then refined with a margin loss on $d_{\text{fib}}$ that pulls paraphrase pairs together and pushes PAWS non-paraphrases apart, with every pair weighted by $\psi_k$.
We add a gluing regularizer. It penalizes the principal-angle mismatch between $P_j$ and $P_k$, weighted by how much the two charts overlap on training data. This is the analogue of smooth transition maps: neighboring charts should agree about rewording where both are responsible.
The chart count $K$, rank $r$ and temperature $\tau$ are chosen on a held-out development split.
Use it as a metric. The trained distance is used in two places. In pairwise equivalence scoring, a threshold on $d_{\text{fib}}$ is set on development data. In diversity scoring, we replace the embedding kernel in the Vendi score, and the Chamfer distance, with kernels computed from $d_{\text{fib}}$.
Baselines
Naive reference points. Two naive references frame the problem. The first is lexical overlap: token F1 and BLEU between the two sentences, thresholded on development data. On PAWS this should sit near chance or below, by construction of the dataset. The second is cosine distance on the same frozen encoder with no training. This isolates what the learned charts add over the raw embedding space. Both use identical inputs and development-set thresholding.
Strongest existing method. The strongest competitor is BERT fine-tuned end to end on PAWS plus QQP, as in PAWS. That paper reports 85% accuracy on the PAWS challenge set after adding 12,000 PAWS training examples. It is the strongest competitor because it attacks exactly the high-overlap failure with the same supervision we use, and it updates every encoder weight.
For fairness we will reproduce it on the same training pairs we give the atlas. It gets the same number of hyperparameter configurations, and it is evaluated on the same splits. We will report our reproduced number alongside the cited 85%.
Beating it on in-domain PAWS-Wiki accuracy is not required for success. The decisive comparisons are two. On out-of-domain transfer (training on PAWS-Wiki and testing on PAWS-QQP, and the reverse), the atlas should be higher. In domain, the atlas should come within a few points while training orders of magnitude fewer parameters on a frozen encoder.
Learned metrics. BERTScore and BLEURT are run off the shelf as the learned-metric baselines. BERTScore reports robustness on adversarial paraphrase detection, but the file gives no accuracy figure: no reported result found on Lacuna for BERTScore accuracy on PAWS. For BLEURT on PAWS, likewise no reported result found on Lacuna.
Diversity. The baseline is VS-Embed-0.5 and Chamfer distance over SimCSE, from Evaluating the Evaluation of Diversity in Commonsense Generation. That study reports over 80% accuracy for these metrics against the GPT-4o-annotated preferences on CommonGen, ComVE and DimonGen. It reports roughly 27% for form-based metrics under word shuffling.
Ablations. Three ablations isolate the key ingredient. The first uses a single global projector ($K=1$) trained with the identical loss and budget; this isolates locality. The second uses hard k-means charts with no overlap and no gluing term; this isolates the partition of unity. The third uses local projectors initialized randomly rather than from paraphrase principal directions; this checks that the fiber estimate, not merely extra capacity, carries the gain.
The atlas counts as working only if it beats the $K=1$ projector by a margin exceeding seed variance across five seeds.
Why not just…?
Why not just fine-tune the encoder on PAWS? This is the real threat. The 85% result shows it works in domain. The PAWS paper's own numbers also show that encoders trained without adversarial data learn the overlap shortcut, which suggests the fix is tied to the training distribution.
Our experiment settles the question by cross-domain transfer between PAWS-Wiki and PAWS-QQP, and by the diversity task. On that task, a PAWS-fine-tuned classifier gives no usable distance. If fine-tuning transfers as well as the atlas, we will report that the atlas's advantage is only cost and interpretability.
Why not just learn one global projection or metric? This is the simplest geometric alternative, and it is exactly the $K=1$ ablation. The principal-angle diagnostic is designed to show in advance whether it can suffice. If rewording directions are shared across regions, the global projection should match the atlas, and we will say so rather than defend the charts.
Why not just use BLEURT or an LLM judge? Both are plausibly more accurate on hard cases. However, BLEURT needs a full cross-encoder pass per pair, and LLM judging costs about 140 minutes and $120 per 10,000 examples, per PEDANTS. Neither can serve as the pairwise kernel inside a Vendi-style diversity score over large sample sets.
We will report BLEURT's accuracy on PAWS as an upper reference. We will position the atlas as a cheap, embedding-level distance, not as a replacement for judges.
Why not just a mixture of per-cluster classifiers? Hard clusters are the second ablation. Their failure mode should appear near cluster boundaries, so we will report accuracy stratified by chart-assignment entropy. If overlap matters, the gap between the atlas and hard clusters should concentrate on pairs whose midpoint $m$ has high-entropy $\psi$.
Proposed Experiments
The first decisive check is the principal-angle diagnostic on QQP and PAWS-Wiki embeddings. Its outcome determines whether the rest of the program proceeds.
The second is paraphrase identification on PAWS-Wiki and PAWS-QQP. We report accuracy and area under the ROC curve for all baselines and ablations, over five seeds, both in domain and cross domain.
The third is diversity meta-evaluation on CommonGen, ComVE and DimonGen, following the protocol of Evaluating the Evaluation of Diversity in Commonsense Generation. We add one new stress set: output sets inflated by backtranslated paraphrases of a single sentence, using the backtranslation step from PAWS. These sets should score as low diversity. Word-shuffled sets should also score as low diversity, which is the case where form-based metrics fail.
Success means three things. First, the atlas matches or exceeds VS-Embed's reported above-80% accuracy on the original sets. Second, it clearly beats VS-Embed on the paraphrase-inflated sets. Third, it does not regress on shuffled sets.
As a secondary check, we compute segment-level Kendall's $\tau$ on WMT17 against BERTScore and BLEURT. No per-pair coefficients are reported in the file, so all three will be run ourselves.
Finally, we run a qualitative analysis of the learned charts. For each chart we inspect which PAWS failure types it absorbs, for example swapped agents, numbers, or named entities. This tells us whether the charts align with interpretable regions of meaning or are arbitrary partitions.
References
- Learning a Manifold as an Atlas
- Atlas Generative Models and Geodesic Interpolation
- Implications of data topology for deep generative models
- Bundle Networks: Fiber Bundles, Local Trivializations, and a Generative Approach to Exploring Many-to-one Maps
- Unsupervised Sentence-embeddings by Manifold Approximation and Projection
- PAWS: Paraphrase Adversaries from Word Scrambling
- BERTScore: Evaluating Text Generation with BERT
- BLEURT: Learning Robust Metrics for Text Generation
- Evaluating the Evaluation of Diversity in Commonsense Generation
- PEDANTS: Cheap but Effective and Interpretable Answer Equivalence