TL;DR
This paper introduces the Attention-Driven Variational Autoencoder (ADVAE), an unsupervised framework that decomposes sentence representations into distinct latent variables aligned with core syntactic roles (such as subjects, verbs, and objects). By leveraging cross-attention mechanisms inspired by machine translation, the model allows researchers to inspect and independently edit specific linguistic components of a sentence without requiring syntactic tree supervision during training.
What Problem Does This Paper Address?
In NLP representation learning, disentanglement has largely focused on separating high-level style from content (e.g., sentiment transfer) or separating general syntax from semantics, often relying on supervised parse trees or parallel paraphrase datasets.
However, decomposing the internal content of a sentence without supervision remains an open challenge. Specifically, human language conveys meaning through predicative structures: who (subject) did what (verb) to whom (object). Prior sentence-level VAEs either entangle these core semantic arguments into an uninterpretable single vector or require explicit supervisory signals to separate them. The authors ask: Can an unsupervised neural model naturally factorize sentence content into independent latent representations corresponding to basic syntactic roles?
Methods
The authors propose the Attention-Driven Variational Autoencoder (ADVAE):
- Inference Model (Encoder): Rather than pooling token embeddings into a single vector, the encoder feeds the sentence tokens and a fixed number ($N_Z$) of learned, input-independent queries through Transformer cross-attention layers. Each query extracts information from specific spans to parameterize the mean $\mu_i$ and standard deviation $\sigma_i$ of an independent latent Gaussian variable $z_i$.
- Generative Model (Decoder): Latent variables $z_1, \dots, z_{N_Z}$ are concatenated with learned identifier embeddings and serve as memory keys/values. An autoregressive Transformer decoder reconstructs the sentence by cross-attending to these latents.
- Training Objective: Standard $\beta$-VAE loss, balancing sentence reconstruction against KL divergence to prevent posterior collapse while promoting factor independence.
Input Sentence ──► Transformer Encoder
│ (Cross-Attention)
Learnable Queries ──────────┴─► Latents (z_subj, z_verb, z_obj, ...)
│
▼ (Cross-Attention)
Generated Text ◄── Transformer Decoder ◄── Autoregressive Target
Novel Evaluation Protocol
To measure disentanglement over variable-length text spans without ground-truth latent labels:
- Syntactic Span Extraction: An off-the-shelf parser (spaCy / OntoNotes) extracts spans for verbal roots (
verb), nominal subjects (subj), direct objects (dobj), and prepositional objects (pobj). - Encoder Disentanglement ($\mathbb{D}_{enc}$): Evaluates whether each latent query's attention weights peak predictably over tokens belonging to a specific syntactic role.
- Decoder Disentanglement ($\mathbb{D}_{dec}$): Resamples a single latent variable $z_i$, decodes the text, and quantifies whether changes in the generated sentence are strictly isolated to the corresponding role's textual span.
Datasets
- SNLI Premises: The primary benchmark consists of raw premise sentences from the Stanford Natural Language Inference dataset (~90k train, 5k dev, 5k test). These sentences average ~8.9 tokens with relatively consistent predicative structures.
- Yelp Reviews: Evaluated in the appendix as a more structurally diverse, noisy dataset of user-generated content (~8.9 tokens on average).
Key Insights and Ideas
- Cross-Attention as an Inductive Bottleneck: Unlike standard Transformer VAEs that pool representations into a single global vector, cross-attention with fixed queries provides a natural inductive bias that clusters tokens according to dependency subtrees.
- Early Layers Specialize in Syntax: Inspecting layer-wise encoder attention revealed that the first layer exhibits the sharpest, sparsest alignment to syntactic roles, aligning with linguistic probing literature showing that early Transformer layers process local syntax.
- The "Co-Adaptation" Phenomenon: Latents are modeled as statistically independent priors ($p(z) = \prod p(z_i)$), but language is inherently constrained (verbs dictate transitivity and semantically plausible arguments). When an object or verb is resampled in isolation, the decoder occasionally modifies adjacent roles to maintain grammatical and semantic plausibility.
Key Results
- Role Separation ($N_\Gamma$): On SNLI, ADVAE ($N_Z = 4$) consistently maps distinct syntactic roles to separate latents ($N_{\Gamma} = 3.0$), successfully isolating
subj,verb, and object roles (dobj/pobjfrequently share a latent). In contrast, Sequence VAEs conflate multiple roles into single dimensions ($N_{\Gamma}^{dec} = 1.40 - 2.40$). - Disentanglement Concentration ($\mathbb{D}$): ADVAE outperforms standard Transformer VAEs by a wide margin on decoder concentration ($\mathbb{D}_{dec} \approx 0.71\text{--}0.98$ vs. $0.11\text{--}0.12$).
- Controllable Text Editing: By intervening on a single latent variable, ADVAE can swap or resample individual roles (e.g., swapping "a woman is talking on a train" to "people are talking on a train") while leaving the rest of the sentence intact.
Limitations and Future Work
- Handling Complex and Irregular Syntax: On noisy text (Yelp) featuring non-verbal roots, fragments, or coordinating conjunctions, disentanglement scores dropped significantly.
- Object Grouping: The model struggled to split direct objects (
dobj) and prepositional objects (pobj) into separate latents, grouping them into a generalized predicate-argument slot. - Independent vs. Structured Priors: Replacing the independent Gaussian prior with a hierarchical prior yielded training instability and higher variance, indicating that better learning algorithms are needed to model dependencies among arguments without sacrificing disentanglement.
Summary updated