Shuffle-Invariance as a Free Teacher for Separating Meaning from Grammar in Bidirectional Encoders

Word order carries most of a sentence's grammar and surprisingly little of its topic. If you shuffle the words of "the researchers criticized the proposal," a reader can no longer recover who did what in which tense, but can still tell the sentence is about researchers, criticism and a proposal. We propose to turn this asymmetry into a training signal. A bidirectional encoder is a model that builds each word's representation from the words on both its left and its right. We train such an encoder so that every token's vector is split into two parts. The "meaning" part must come out the same whether the sentence is read in order or shuffled. The "grammar" part must be able to recover where the token sat in the original order. No parse trees, paraphrase pairs or lexicons are required, only raw text and a random permutation.

This matters because every current method that cleanly pulls syntax apart from semantics works on one vector per sentence and depends on outside supervision. Methods that work on one vector per word, by contrast, cannot use context at all. Neither gives the per-token, context-aware split that tagging, retrieval and controlled editing actually use. If the split works, one encoder could hold a single representation that sits at both ends of the trade-off. Artetxe et al. showed that a single static space can only occupy one point on that trade-off.

Core Claim

Our claim is that order-invariance and order-recoverability are complementary self-supervised targets. Imposed on two halves of each token's bidirectional hidden state, they produce two subspaces. The first should match or beat parse-supervised sentence disentanglement on semantic similarity. The second should match order-trained word representations on part-of-speech tagging and parsing probes. Together the two should trace a better trade-off frontier than any single-space re-weighting. We also claim that bidirectionality is necessary rather than incidental. A left-to-right model cannot give a token a representation built from the whole sentence, so it cannot be asked to be invariant to reordering that whole sentence. We test this directly with a unidirectional ablation.

What exists and what is missing

Three findings motivate the proposal. Uncovering Divergent Linguistic Information in Word Embeddings re-weights the eigen-directions of static word vectors with a single exponent $\alpha$. That one dial raised GloVe's SimLex-999 Spearman correlation from 40.70 to 51.54, but the best setting for semantics and the best setting for syntax diverged. The paper therefore shows a real trade-off inside one space, using only a global, context-free warp. Word Ordering as Unsupervised Learning Towards Syntactically Plausible Word Representations trains embeddings by reconstructing word order from a shuffled bag. It reached 97.04% WSJ part-of-speech accuracy, above Skip-gram's 96.76%, but needed an explicit anchor to Skip-gram vectors to stop meaning from decaying. We borrow its signal, order recovery, but assign it to only half of each token vector. The other half is protected by a matching invariance objective rather than an external anchor. Syntactic Structure Distillation Pretraining For Bidirectional Encoders distills an RNN Grammar into BERT. This gave large structural gains, including a 21.57% relative error reduction on CCG supertagging, but mixed GLUE results and pretraining about 5× slower. We read this as the trade-off reappearing when syntax is pushed into one shared space. Our design keeps syntax out of the space that carries meaning.

Three papers supply the closest methods, and all of them work at the sentence level. A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations (VGVAE) splits a sentence into a semantic and a syntactic latent using paraphrase pairs and a word-position loss. It reports 72.9 Pearson on the STS Benchmark and a k-nearest-neighbour constituency F1 of 38.8 for the syntactic latent versus 25.3 for the semantic one. A Deep Decomposable Model for Disentangling Syntax and Semantics (DecVAE) adds a total-correlation independence penalty and reaches 73.91 on STS and 41.5 parsing F1. Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models (ParaBART) feeds a parse tree to a separate encoder and adversarially scrubs syntax from the semantic one. It reaches a 75.9 average Pearson across SemEval STS 2012–2016 and the STS Benchmark, against 70.6 for Sentence-BERT. All three need paraphrase pairs, and ParaBART also needs parse trees. All three compress the sentence into one bottleneck and leave token states unexamined. We keep VGVAE's position-loss idea and DecVAE's independence penalty, apply them per token, and replace paraphrase supervision with shuffle-invariance.

One result constrains the design. Cooperative Learning of Disjoint Syntax and Semantics shows that when two modules share a loss, the faster-learning one makes the other irrelevant. Its latent-tree parsers collapsed to trivial trees, and ListOps accuracy fell to about 60% without pacing control. We therefore give each half its own target rather than letting both serve one reconstruction loss, and we monitor whether either half becomes predictable from the other.

Research Program

Take a bidirectional encoder producing a hidden state $h_t$ for token $t$ in sentence $x$. Two linear maps split it into a meaning part $s_t = A h_t$ and a grammar part $g_t = B h_t$. Let $\pi(x)$ be a random permutation of the sentence. The training objective is

$$ \mathcal{L} = \mathcal{L}_{\text{MLM}} + \lambda_1 \, \mathcal{L}_{\text{inv}}(s) + \lambda_2 \, \mathcal{L}_{\text{pos}}(g) + \lambda_3 \, \mathcal{L}_{\text{dec}}(s, g). $$

Each term has a specific job. $\mathcal{L}_{\text{MLM}}$ is the usual masked-word loss, which keeps the full state useful. $\mathcal{L}_{\text{inv}}$ is a contrastive loss that pulls $s_t(x)$ toward the same token's $s_t(\pi(x))$ and pushes it away from other tokens. $\mathcal{L}_{\text{pos}}$ asks a small head on $g_t$ to predict the token's original relative position, or the identity of its original left and right neighbours, from the shuffled input. $\mathcal{L}_{\text{dec}}$ penalizes cross-covariance between $s$ and $g$, a cheap stand-in for DecVAE's total-correlation term.

The invariance loss has a trivial solution: $s_t$ could encode only the word's identity and ignore context. We block this by drawing hard negatives from the same word type in other sentences. To separate "bank" in a finance sentence from "bank" in a river sentence, $s_t$ must use the bag of context words, which is exactly the order-free information we want it to hold.

The program has three stages. First, a cheap diagnostic on existing bidirectional models answers whether the split already exists. We take a BiLSTM and a bidirectional linear recurrent encoder in the style of BLUR, which runs forward and backward parallel scans and fuses them. We probe their forward states, backward states and fused states separately on part-of-speech tagging and on a shuffle-invariance score. This tests the unexamined assumption, noted around both BLUR and DeltaFlow, that the two scan directions do interchangeable work. Second, we train the split objective in two settings: continued training of a pretrained Transformer encoder (BERT-base), and from-scratch training of a BLUR-style linear recurrent encoder of matched size on OpenWebText. The second setting answers whether linear-time bidirectional recurrence can hold two clean subspaces, which no paper in the file has probed on linguistic tasks. Third, we test whether the split helps a downstream use where the trade-off bites. In iterative bidirectional infilling in the style of SENSESHIFT, we steer the edit only through $s$ and freeze $g$. Success would mean semantic steering without loss of contextual fit.

Baselines

The naive reference point is the encoder's raw hidden states under the same continued-training budget with MLM alone. This answers whether the extra objectives buy anything beyond more training steps. Raw states are a weak reference on similarity: VGVAE reports raw BERT at 4.5 STS Pearson, though that number reflects pooling rather than a well-tuned baseline. We will also report mean-pooled raw states so the reference point is fair.

The strongest single-space competitor is the Artetxe et al. eigen-reweighting, which we apply per layer to contextual token states rather than to static vectors. We sweep $\alpha$ on development data and pick separate settings for the semantic and syntactic evaluations, giving this baseline every advantage. It is the most direct threat because it needs no training. If a pair of $\alpha$ settings matches our $s$ and $g$ on their respective tasks, our objective is unnecessary.

The strongest disentanglement method is ParaBART, with a reported 75.9 average Pearson on SemEval STS 2012–2016 plus the STS Benchmark. On the syntax side the strongest is DecVAE, with 41.5 nearest-neighbour constituency F1 on the Penn Treebank. We will rerun ParaBART from released code on the same base encoder size and evaluate it on the identical STS suite. For DecVAE we will report the published number alongside our own rerun, so that differences in evaluation protocol are visible.

Beating the baselines means two things. Our $s$ should come within one Pearson point of ParaBART on the STS average without parse trees or paraphrase pairs, and exceed it on QQP-Hard. Our $g$ should exceed DecVAE's 41.5 under the same nearest-neighbour protocol. We found no reported numeric result on Lacuna for ParaBART on QQP-Hard, so that comparison is relative to our own rerun.

On the syntax side we add a word-ordering baseline. WON's 97.04% WSJ tagging is the reference, and we also run an order-recovery-only variant of our model with no invariance half. We further report the syntactic-distillation result of Kuncoro et al. as an upper reference for parse-supervised structure, without retraining it.

Four ablations isolate the key ingredient. We remove $\mathcal{L}_{\text{inv}}$, remove $\mathcal{L}_{\text{pos}}$, and remove $\mathcal{L}_{\text{dec}}$. Most importantly, we replace the bidirectional encoder with a left-to-right one of equal size. Our claim predicts that this last change breaks the invariance half, because a left-to-right state never sees the whole bag. All variants share data, steps, parameter count and a fixed hyperparameter search of equal size.

Why not just…?

Why not just apply the Artetxe re-weighting to contextual states and skip training? This is the most serious threat, so it is a main baseline rather than a footnote. The paper's own evidence shows that one global warp picks a point on the trade-off rather than escaping it. Using it twice with two $\alpha$ values gives two views, but both are linear functions of the same entangled state, applied uniformly across all contexts. If our frontier plot, with STS on one axis and parsing F1 on the other, shows the two-$\alpha$ curve reaching our points, we will conclude that the split was latent all along and report that as the finding.

Why not just take lower layers for syntax and upper layers for semantics? Layer selection is a cheap, common heuristic, and we include every layer of the raw encoder in the frontier plot. The Kuncoro et al. results suggest it is not enough: BERT's own layers left a 21.57% CCG error gap that explicit structure closed. The ParaBART authors also argue that raw encoders confuse syntactically similar but semantically different sentences. Still, if some layer pair matches our subspaces, the claim fails, and the experiment shows this directly.

Why not just use ParaBART-style parse supervision, which already works? It does work, and it is our accuracy target. However, it needs a constituency parser at training time and produces only sentence vectors. Our proposal is worth doing only if it gets close on sentence similarity while adding token-level subspaces and dropping the parser. Beyond the headline numbers, we will therefore report a fully unsupervised training run on a language without a treebank, Korean or German raw text, evaluated with part-of-speech probes. In that setting ParaBART cannot be trained at all.

Why not just train a word-ordering model and anchor it to Skip-gram, as WON did? WON's anchor protects word-type meaning, but cannot protect context-dependent meaning, because Skip-gram vectors have no context. Our invariance half is contextual by construction. The test is the same-word-type hard negatives: if $s_t$ separates senses of one word across sentences while WON-style anchored vectors cannot, the difference is real.

Proposed Experiments

The semantic evaluations are the STS Benchmark and SemEval STS 2012–2016, scored by Pearson correlation on mean-pooled $s$, and QQP-Hard for paraphrases with divergent syntax. At the word level, SimLex-999 uses $s$ averaged over contexts, and we report the semantic half of the Google analogy set. The syntactic evaluations are WSJ part-of-speech tagging with a linear probe on $g$, nearest-neighbour Penn Treebank constituency F1 following the VGVAE and DecVAE protocol, CCG supertagging, the syntactic half of the Google analogy set, and the SentEval TreeDepth and BShift probes. Cross-probing measures the split: we report how well $s$ does on syntax and how well $g$ does on semantics, where lower is better. VGVAE's 38.8 versus 25.3 gap is the template for this measurement.

The decisive check is the frontier plot. Every method, layer and $\alpha$ setting becomes a point in the plane of STS average against Penn Treebank nearest-neighbour F1. The proposal succeeds if our $(s, g)$ pair lies outside the region reachable by any single representation of the raw encoder.

The second check is the left-to-right ablation. Our explanation predicts a large drop in STS for $s$ but only a small drop in tagging for $g$. A symmetric drop would mean bidirectionality is merely extra capacity.

A third check is honest about a cost. Our $s$ ignores order by design, so it will call "the dog bit the man" and "the man bit the dog" equivalent. We will build a small role-swap test from STS sentences and report it for $s$ alone and for $[s; g]$. We expect $s$ to fail that test by design and the concatenation to recover role information. This failure mode is exactly the false-positive problem ParaBART names.

Finally, on TinyStories infilling, we compare steering through $s$ alone against steering the full state. We measure the SENSESHIFT sentiment error and contextual-fitness score, with SENSESHIFT's reported $\Delta_s = 0.20$ and 0.79 sentiment correlation as the reference point. If steering through $s$ alone matches the full state on sentiment while improving fitness, the split pays off beyond probing.