Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs

Ghazi Felhi, Joseph Le Roux, Djamé Seddah

Abstract

We propose a generative model for text generation, which exhibits disentangled latent representations of syntax and semantics. Contrary to previous work, this model does not need syntactic information such as constituency parses, or semantic information such as paraphrase pairs. Our model relies solely on the inductive bias found in attention-based architectures such as Transformers. In the attention of Transformers, keys handle information selection while values specify what information is conveyed. Our model, dubbed QKVAE, uses Attention in its decoder to read latent variables where one latent variable infers keys while another infers values. We run experiments on latent representations and experiments on syntax/semantics transfer which show that QKVAE displays clear signs of disentangled syntax and semantics. We also show that our model displays competitive syntax transfer capabilities when compared to supervised models and that comparable supervised models need a fairly large amount of data (more than 50K samples) to outperform it on both syntactic and semantic transfer. The code for our experiments is publicly available.

Introduction

Disentangling syntax (structure) from semantics (meaning) is a fundamental challenge in Natural Language Processing. Successful disentanglement enables controlled text generation, such as rewriting a sentence in a different voice while preserving its core message. Traditionally, this has required supervised learning on large datasets of paraphrase pairs or syntactic parses. However, such data is expensive and often domain-specific. This paper introduces QKVAE, a model that achieves this separation without any labeled data by leveraging the inherent architectural biases of the Transformer.

Methodology: The QKVAE Architecture

The core idea behind QKVAE (Query-Key-Value VAE) is to exploit the functional roles of the Transformer's attention mechanism. In standard attention, Keys ($K$) determine where to look, while Values ($V$) determine what content is retrieved. The authors hypothesize that this mapping naturally corresponds to the linguistic distinction between syntax and semantics.

The model employs a split latent space:

  1. Semantic Latent ($z^{sem}$): Inferred from the input to generate the Values in the decoder.
  2. Syntactic Latent ($z^{syn}$): Inferred to generate the Keys in the decoder.

By forcing these two types of information through separate bottlenecks, the architecture "traps" structural information in the key-generating latent and meaning in the value-generating latent.

Syntactic control through Keys: Applying different structural keys (k1, k2) to the same semantic values (v) results in different syntactic realizations.

Experimental Evaluation

The authors evaluated QKVAE on the ParaNMT dataset. While supervised baselines (VGVAE, SynPG) used paraphrase pairs, QKVAE was trained in a purely unsupervised fashion using individual sentences. To prevent "posterior collapse"—a common VAE issue where the model ignores latent variables—the authors utilized a specific annealing schedule, prioritizing the learning of semantic variables first.

Key findings include:

  • Embedding Specialization: Quantitative analysis confirmed that $z^{syn}$ effectively clusters sentences by their syntactic tree structure, while $z^{sem}$ clusters by semantic similarity.
  • Syntactic Transfer: QKVAE demonstrated superior transfer capabilities compared to other unsupervised methods like ADVAE and remained competitive with supervised models.

Results and the "50K Rule"

A significant finding of this research is the data efficiency of the unsupervised approach. The authors compared QKVAE against the supervised VGVAE model trained on varying amounts of labeled data.

Performance comparison: QKVAE (unsupervised) vs. VGVAE (supervised) across different training sizes. QKVAE remains competitive with supervised models until they reach approximately 50K-100K samples.

As shown in the figure above, the supervised model requires between 50,000 and 100,000 labeled samples to consistently outperform the unsupervised QKVAE in both semantic preservation (measured by PB) and syntactic transfer (measured by STED). This suggests that for many low-resource applications, unsupervised inductive bias is a viable and powerful alternative to costly data labeling.

Conclusion

QKVAE demonstrates that architectural design can substitute for explicit supervision in complex linguistic tasks. While the current latent variables are disentangled, they are not yet fully interpretable in human-linguistic terms (e.g., a specific dimension for "passive voice"). Future work will focus on making these representations more steerable and testing the model's robustness across diverse, morphologically rich languages.

Summary updated

Abstract

We propose a generative model for text generation, which exhibits disentangled latent representations of syntax and semantics. Contrary to previous work, this model does not need syntactic information such as constituency parses, or semantic information such as paraphrase pairs. Our model relies solely on the inductive bias found in attention-based architectures such as Transformers. In the attention of Transformers, keys handle information selection while values specify what information is conveyed. Our model, dubbed QKVAE, uses Attention in its decoder to read latent variables where one latent variable infers keys while another infers values. We run experiments on latent representations and experiments on syntax/semantics transfer which show that QKVAE displays clear signs of disentangled syntax and semantics. We also show that our model displays competitive syntax transfer capabilities when compared to supervised models and that comparable supervised models need a fairly large amount of data (more than 50K samples) to outperform it on both syntactic and semantic transfer. The code for our experiments is publicly available.
BibTeX
@inproceedings{felhi2022exploiting,
  author = {Ghazi Felhi and Joseph Le Roux and Djamé Seddah},
  title = {{Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=W6MZdVMbFX},
}
Figure