Abstract
Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects, ...) and their realizations are essential markers since they can be understood as a decomposition of predicative structures and thus the meaning of sentences. Starting from a deep probabilistic generative model with attention, we measure the interaction between latent variables and realizations of syntactic roles and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose is an Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAEs enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on latent variable perturbations for the decoder. Our experiments on raw English text from the SNLI dataset show that: i) disentanglement of syntactic roles can be induced without supervision, ii) ADVAE separates syntactic roles better than classical sequence VAEs and Transformer VAEs, iii) realizations of syntactic roles can be separately modified in sentences by mere intervention on the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation.
Introduction
In Natural Language Processing (NLP), "disentanglement" typically refers to the separation of high-level attributes, such as style or sentiment, from the core content of a sentence. However, disentangling the internal structure of the content itself—the fundamental "who did what to whom"—remains a significant challenge. Most existing methods for structural disentanglement rely on supervision, requiring pre-parsed trees or labeled data to define syntactic roles like "subject" or "object."
This research addresses the challenge of unsupervised content disentanglement. The goal is to develop a model that can analyze raw text and naturally discover that sentences are composed of independent building blocks (syntactic roles) that can be manipulated individually without human-labeled training data.
Methodology: The Attention-Driven VAE (ADVAE)
The authors introduce the Attention-Driven Variational Autoencoder (ADVAE). Its architecture is inspired by Transformer-based machine translation but modified to facilitate disentanglement through latent variable specialization.
Architecture Components
- The Encoder: Unlike standard VAEs that encode a sentence into a single vector, ADVAE uses a fixed number of independent latent variables ($N_Z$). It employs Cross-Attention to allow these latent variables to "attend" to specific spans of the input text. This mechanism incentivizes each latent variable to specialize in a specific part of the sentence structure.
- The Decoder: An autoregressive Transformer decoder reconstructs the sentence based on these $N_Z$ variables.
- Evaluation Protocol: To measure success without ground-truth labels during training, the authors use an off-the-shelf dependency parser solely for evaluation. They assess whether the model's internal attention (encoder side) and its response to latent perturbations (decoder side) align with real-world syntactic roles. For example, if modifying latent variable #2 primarily changes the subject of the generated sentence, disentanglement is achieved.
Experimental Setup
The model was evaluated on two primary datasets:
- SNLI (Stanford Natural Language Inference): Chosen for its relatively short and structurally regular sentences, providing an ideal environment to verify the capture of core syntactic roles.
- Yelp Reviews: Used to test robustness on user-generated content, which features more diverse and irregular sentence structures.
Key Results and Insights
Role Separation and Alignment
ADVAE successfully separated an average of 3 out of 4 core syntactic roles (subject, verb, direct object, prepositional object) into distinct variables. As shown in the alignment heatmaps, specific latent variables show high sensitivity to specific syntactic roles.
Performance and Controllability
- Superiority over Baselines: ADVAE significantly outperformed standard Sequence VAEs and Transformer VAEs. While standard VAEs often conflated subjects and verbs into a single variable, ADVAE maintained their distinction.
- Controllability: The model demonstrated "role swapping" capabilities. Qualitatively, resampling the "subject" variable allowed for changing the subject of a generated sentence (e.g., from "a woman" to "people") while preserving the rest of the sentence's action and context.
- The "Co-adaptation" Hurdle: The researchers noted that the decoder often tries to maintain grammatical coherence across variables. For instance, changing a subject from singular to plural might cause the decoder to automatically adjust the verb tense. While this improves sentence quality, it represents a form of "co-adaptation" that complicates perfect mathematical disentanglement.
Conclusion and Future Work
ADVAE represents a significant step toward unsupervised controllable content generation. By leveraging cross-attention, the model learns to map linguistic "slots" to independent latent variables.
However, limitations remain. Performance decreases as sentence complexity increases (as seen in the Yelp dataset), and the current model assumes syntactic roles are entirely independent—an assumption that conflicts with the natural dependencies of human language. Future research will explore hierarchical or structured latent variables to better capture these linguistic nuances while maintaining the benefits of unsupervised learning.
Summary updated