Abstract
We propose a generative model for text generation that exhibits disentangled latent representations of syntax and semantics without requiring syntactic labels or paraphrase pairs. Our model, QKVAE, relies on the inherent inductive bias of the Transformer architecture. By mapping syntactic roles to "keys" and semantic content to "values" within the attention mechanism, the model naturally segregates these linguistic features. Experimental results demonstrate that QKVAE achieves clear disentanglement and competitive syntax transfer capabilities compared to supervised models. Notably, supervised counterparts require over 50,000 labeled samples to outperform QKVAE, highlighting the efficiency of architecture-driven inductive bias.
Introduction
In Natural Language Processing, "disentanglement" refers to the creation of neural representations where specific components of a latent code correspond to specific features of the data—most notably, separating what a sentence says (semantics) from how it is structured (syntax). Traditionally, achieving this separation has required heavy supervision, such as human-annotated parse trees or manually curated paraphrase pairs. Such requirements limit the scalability of these models to new languages or domains where such data is scarce.
This paper introduces QKVAE (Query-Key-Value Variational Autoencoder), a model that achieves disentanglement in a purely unsupervised manner. Instead of external labels, QKVAE exploits the structural properties of the Transformer’s attention mechanism to naturally segregate syntactic and semantic information.
Methodology: The QKVAE Framework
The core innovation of QKVAE lies in the mapping of linguistic components to the mathematical components of the Transformer's attention mechanism, building upon the Attention Driven VAE (ADVAE) framework.
The Key/Value Analogy
In standard Transformer attention, keys ($K$) determine which information to select, while values ($V$) provide the actual content. The authors argue that this is a mathematical analogy for language: syntax (the "slots" or "roles" in a sentence) acts like keys, and semantics (the actual words filling those roles) acts like values.
Dual Latent Representations
The QKVAE encoder produces two distinct latent vectors:
- $z^{syn}$ (Syntactic Latent): Dedicated to inferring the keys ($K$) in the decoder's cross-attention.
- $z^{sem}$ (Semantic Latent): Dedicated to inferring the values ($V$).
During text generation, the decoder's cross-attention mechanism uses these separate latent variables to reconstruct the sentence. By forcing the model to use one variable for "where to look" and another for "what to say," the architecture naturally pushes syntactic information into $z^{syn}$ and semantic information into $z^{sem}$.
Optimization
To prevent "posterior collapse"—a common issue where the model ignores latent variables—the authors utilize a $\beta$-VAE objective. This includes a specific weighting for the two variables and a staggered KL-annealing schedule to ensure both latent spaces remain informative and regularized.
Experimental Setup
The model was trained in a purely unsupervised fashion on approximately 493,000 reference sentences from the ParaNMT dataset. Evaluation was conducted using a curated test set of 800 triplets, each consisting of a target sentence, a semantic source (paraphrase), and a syntactic source (structurally similar sentence).
Key Results and Insights
Embedding Specialization
Quantitative tests confirmed that the latent spaces successfully specialized. The syntactic latent variable ($z^{syn}$) was significantly more effective at clustering sentences by their Part-of-Speech (PoS) tags, while the semantic variable ($z^{sem}$) clustered sentences based on their underlying meaning. This confirms that the architecture-driven bias successfully forced the desired disentanglement.
Syntax Transfer and the "50K Rule"
QKVAE proved highly effective at style transfer—taking the structure of one sentence and the meaning of another. It performed competitively with SynPG, a supervised model, on several syntactic matching metrics (TMA2/TMA3).
A striking finding is the efficiency gap: the researchers demonstrated that a supervised model requires more than 50,000 labeled samples to outperform the unsupervised QKVAE on transfer metrics. This suggests that the inductive bias provided by the $Q, K, V$ structure is a powerful substitute for large-scale human annotation.
Qualitative Success
The model successfully handled complex transformations, such as converting a sentence from active to passive voice or maintaining subject-verb inversion, simply by swapping the $z^{syn}$ variable. Furthermore, because it is a VAE, the latent space is continuous, allowing for smooth interpolations between sentences that often result in grammatically correct intermediate forms.
Limitations and Future Work
While syntax transfer is robust, semantic transfer is described as "moderate." The model occasionally struggles to maintain exact lexical items from the semantic source, often opting for close synonyms due to the smoothness of the VAE space. This results in lower lexical overlap compared to some supervised methods.
Future research will explore applying the QKVAE Key/Value split to non-textual data, such as images, to determine if it can disentangle visual "objects" from their "positions." Additionally, the authors aim to test the model on more linguistically diverse languages to verify the universality of this inductive bias.
Conclusion
QKVAE demonstrates that the internal mechanics of the Transformer architecture can be harnessed to achieve sophisticated disentanglement without the need for labeled data. By aligning the mathematical roles of keys and values with the linguistic roles of syntax and semantics, the model provides a robust and efficient framework for unsupervised representation learning.
Summary updated