Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks

Yingji Zhang, Danilo Carvalho, André FreitasACL 2024 · DOI · OpenReview

TL;DR

This paper introduces a novel method for disentangling semantic features within sentences using invertible neural networks (INNs) integrated with transformer-based autoencoders. This approach leads to more interpretable and controllable text generation by creating a latent space where semantic components like predicates and arguments are clearly separated.

What Problem Does This Paper Address?

While disentangling latent spaces for better interpretability and controllable generation has seen success in computer vision (e.g., image disentanglement), it remains a significant challenge in Natural Language Processing (NLP), particularly for sentence-level semantics. Existing methods often focus on specific factors like sentiment or style transfer. This paper tackles the broader problem of sentence semantic disentanglement, aiming to achieve localized control over general semantic features within sentences. The goal is to bridge the gap between the flexibility of distributional neural models and the precision of formal linguistic representations, enabling more granular control over text generation.

Methods

The core of the proposed method involves integrating a flow-based Invertible Neural Network (INN) with a transformer-based language Autoencoder (AE). Here's a breakdown:

  1. Pre-training an Autoencoder: First, a standard AE (specifically, a BERT-GPT2 based model) is pre-trained to learn sentence representations from transformer latent spaces.
  2. Integrating INNs: The AE's learned representations are then fed into an INN. INNs are powerful because they learn a bijective mapping between an observation distribution and a latent distribution. This means you can not only map from data to latent space but also perfectly reconstruct the data from the latent space.
  3. Supervised Training Strategy: The key novelty lies in a supervised training strategy for the INN. This strategy leverages Argument Structure Theory (AST), a linguistic framework that models sentences as predicate-argument structures. The INN is trained to separate these role-content clusters (e.g., "ARG0-animal," "PRED-require") in the latent space. This is achieved by minimizing the distance between points within the same role-content region and maximizing the distance between different regions.
  4. Geometrical Data Augmentation: The authors also propose using geometrical data augmentation techniques (like vector arithmetic and traversal operators) in the latent space to further enhance disentanglement and support controlled sentence generation.

Datasets

The paper primarily uses explanatory sentences from datasets like WorldTree (Jansen et al., 2018a) and EntailmentBank (Dalvi et al., 2021). These datasets are chosen because they are designed for formal/localized/symbolic semantic inference tasks, providing complex yet controlled scenarios for evaluating semantic disentanglement.

Key Insights and Ideas

  • Bridging Formal and Distributional Semantics: The work successfully bridges formal linguistic theories (like AST) with distributional neural representations, offering a more principled way to understand and control sentence meaning.
  • INN for Semantic Disentanglement: INNs, with their bijective mapping property, are shown to be an effective mechanism for creating a smooth, controllable latent space for sentence representations, overcoming limitations of traditional VAEs that approximate priors.
  • AST-Guided Supervision: Using Argument Structure Theory as a guide for supervised training is crucial. It allows the model to learn to separate specific semantic roles and their associated content, leading to a more structured and interpretable latent space.
  • Geometrical Control: The disentangled latent space enables geometrical operations like interpolation for controlled sentence generation, allowing for smoother and more semantically coherent transitions between sentences.

Key Results

  • Improved Latent Space Separability: The cluster-supervised INN approach significantly improves the separability of role-content clusters (e.g., ARGO, ARG1, PRED) in the latent space compared to unsupervised INNs and baseline models like Optimus. This is demonstrated through classification metrics (accuracy, precision, recall, F1 score) where the supervised INN consistently outperforms others.
  • Better Disentanglement: Quantitative evaluations show that the supervised INN achieves better disentanglement for various semantic roles, with improvements ranging from a few percentage points to several points depending on the classifier and role.
  • Stable Invertibility: Both unsupervised and supervised INNs demonstrate stable invertibility, meaning they can reliably reconstruct sentences from their latent representations (invertibility ratios often above 90%).
  • Controlled Generation: Qualitative results from interpolation experiments show that the supervised INN can maintain role-content consistency during generation, leading to more semantically coherent intermediate sentences compared to baselines. For instance, it can fix the predicate-require during interpolation, indicating better control.
  • Data Augmentation Benefits: The proposed data augmentation strategy helps maintain semantic distribution and can further enhance classification performance, suggesting its utility in improving disentanglement.

Limitations and Future Work

  • Safety Guarantees: The paper acknowledges that specific safety guarantees for these disentangled models are not fully established.
  • Unsupervised INN Understanding: Further research is needed to fully understand the semantic distribution and information bottleneck properties of unsupervised INNs in language modeling tasks.
  • Corpus Specificity: The current study focuses on conceptually dense explanatory sentences. Its performance on sentences with complex clausal-phrasal constructions or non-compositional idioms remains to be explored.
  • Future Directions: The authors suggest exploring the safety and control of formal semantic properties in Large Language Models. They also propose adapting methods that model latent vector movement using ordinary differential equations for more complex sentence transformations.

Summary updated

Abstract

Disentangled latent spaces usually have better semantic separability and geometrical properties, which leads to better interpretability and more controllable data generation. While this has been well investigated in Computer Vision, in tasks such as image disentanglement, in the NLP domain, sentence disentanglement is still comparatively under-investigated. Most previous work have concentrated on disentangling task-specific generative factors, such as sentiment, within the context of style transfer. In this work, we focus on a more general form of sentence disentanglement, targeting the localised modification and control of more general sentence semantic features. To achieve this, we contribute to a novel notion of sentence semantic disentanglement and introduce a flow-based invertible neural network (INN) mechanism integrated with a transformer-based language Autoencoder (AE) in order to deliver latent spaces with better separability properties. Experimental results demonstrate that the model can conform the distributed latent space into a better semantically disentangled sentence space, leading to improved language interpretability and controlled generation when compared to the recent state-of-the-art language VAE models.
BibTeX
@inproceedings{zhang2024learning,
  author = {Yingji Zhang and Danilo Carvalho and André Freitas},
  title = {{Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.116},
}
Figure