Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles

Ghazi Felhi, Joseph Le Roux, Djamé Seddah

TL;DR

This paper introduces the Attention-Driven Variational Autoencoder (ADVAE), an unsupervised framework that decomposes sentence representations into distinct latent variables aligned with core syntactic roles (such as subjects, verbs, and objects). By leveraging cross-attention mechanisms inspired by machine translation, the model allows researchers to inspect and independently edit specific linguistic components of a sentence without requiring syntactic tree supervision during training.

What Problem Does This Paper Address?

In NLP representation learning, disentanglement has largely focused on separating high-level style from content (e.g., sentiment transfer) or separating general syntax from semantics, often relying on supervised parse trees or parallel paraphrase datasets.

However, decomposing the internal content of a sentence without supervision remains an open challenge. Specifically, human language conveys meaning through predicative structures: who (subject) did what (verb) to whom (object). Prior sentence-level VAEs either entangle these core semantic arguments into an uninterpretable single vector or require explicit supervisory signals to separate them. The authors ask: Can an unsupervised neural model naturally factorize sentence content into independent latent representations corresponding to basic syntactic roles?

Methods

The authors propose the Attention-Driven Variational Autoencoder (ADVAE):

  • Inference Model (Encoder): Rather than pooling token embeddings into a single vector, the encoder feeds the sentence tokens and a fixed number ($N_Z$) of learned, input-independent queries through Transformer cross-attention layers. Each query extracts information from specific spans to parameterize the mean $\mu_i$ and standard deviation $\sigma_i$ of an independent latent Gaussian variable $z_i$.
  • Generative Model (Decoder): Latent variables $z_1, \dots, z_{N_Z}$ are concatenated with learned identifier embeddings and serve as memory keys/values. An autoregressive Transformer decoder reconstructs the sentence by cross-attending to these latents.
  • Training Objective: Standard $\beta$-VAE loss, balancing sentence reconstruction against KL divergence to prevent posterior collapse while promoting factor independence.
Input Sentence  ──►  Transformer Encoder
                            │ (Cross-Attention)
Learnable Queries ──────────┴─► Latents (z_subj, z_verb, z_obj, ...)
                                         │
                                         ▼ (Cross-Attention)
Generated Text  ◄──  Transformer Decoder ◄── Autoregressive Target

Novel Evaluation Protocol

To measure disentanglement over variable-length text spans without ground-truth latent labels:

  1. Syntactic Span Extraction: An off-the-shelf parser (spaCy / OntoNotes) extracts spans for verbal roots (verb), nominal subjects (subj), direct objects (dobj), and prepositional objects (pobj).
  2. Encoder Disentanglement ($\mathbb{D}_{enc}$): Evaluates whether each latent query's attention weights peak predictably over tokens belonging to a specific syntactic role.
  3. Decoder Disentanglement ($\mathbb{D}_{dec}$): Resamples a single latent variable $z_i$, decodes the text, and quantifies whether changes in the generated sentence are strictly isolated to the corresponding role's textual span.

Datasets

  • SNLI Premises: The primary benchmark consists of raw premise sentences from the Stanford Natural Language Inference dataset (~90k train, 5k dev, 5k test). These sentences average ~8.9 tokens with relatively consistent predicative structures.
  • Yelp Reviews: Evaluated in the appendix as a more structurally diverse, noisy dataset of user-generated content (~8.9 tokens on average).

Key Insights and Ideas

  • Cross-Attention as an Inductive Bottleneck: Unlike standard Transformer VAEs that pool representations into a single global vector, cross-attention with fixed queries provides a natural inductive bias that clusters tokens according to dependency subtrees.
  • Early Layers Specialize in Syntax: Inspecting layer-wise encoder attention revealed that the first layer exhibits the sharpest, sparsest alignment to syntactic roles, aligning with linguistic probing literature showing that early Transformer layers process local syntax.
  • The "Co-Adaptation" Phenomenon: Latents are modeled as statistically independent priors ($p(z) = \prod p(z_i)$), but language is inherently constrained (verbs dictate transitivity and semantically plausible arguments). When an object or verb is resampled in isolation, the decoder occasionally modifies adjacent roles to maintain grammatical and semantic plausibility.

Key Results

  • Role Separation ($N_\Gamma$): On SNLI, ADVAE ($N_Z = 4$) consistently maps distinct syntactic roles to separate latents ($N_{\Gamma} = 3.0$), successfully isolating subj, verb, and object roles (dobj/pobj frequently share a latent). In contrast, Sequence VAEs conflate multiple roles into single dimensions ($N_{\Gamma}^{dec} = 1.40 - 2.40$).
  • Disentanglement Concentration ($\mathbb{D}$): ADVAE outperforms standard Transformer VAEs by a wide margin on decoder concentration ($\mathbb{D}_{dec} \approx 0.71\text{--}0.98$ vs. $0.11\text{--}0.12$).
  • Controllable Text Editing: By intervening on a single latent variable, ADVAE can swap or resample individual roles (e.g., swapping "a woman is talking on a train" to "people are talking on a train") while leaving the rest of the sentence intact.

Limitations and Future Work

  • Handling Complex and Irregular Syntax: On noisy text (Yelp) featuring non-verbal roots, fragments, or coordinating conjunctions, disentanglement scores dropped significantly.
  • Object Grouping: The model struggled to split direct objects (dobj) and prepositional objects (pobj) into separate latents, grouping them into a generalized predicate-argument slot.
  • Independent vs. Structured Priors: Replacing the independent Gaussian prior with a hierarchical prior yielded training instability and higher variance, indicating that better learning algorithms are needed to model dependencies among arguments without sacrificing disentanglement.

Summary updated

Abstract

Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects,$\dots$) and their realizations are essential markers since they can be understood as a decomposition of predicative structures and thus the meaning of sentences. Starting from a deep probabilistic generative model with attention, we measure the interaction between latent variables and realizations of syntactic roles, and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose is an Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAEs enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on disturbing individual latent variables for the decoder. Our experiments on raw English text from the SNLI dataset show that $\textit{i)}$ disentanglement of syntactic roles can be induced without supervision, $\textit{ii)}$ ADVAE separates more syntactic roles than classical sequence VAEs, $\textit{iii)}$ realizations of syntactic roles can be separately modified in sentences by mere intervention on the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation. The code for our work is publicly available.
BibTeX
@inproceedings{felhi2022towards,
  author = {Ghazi Felhi and Joseph Le Roux and Djamé Seddah},
  title = {{Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=fyLvrx9M9YP},
}

Loading related content...

Figure