ESANN 2022 proceedings

Andrea Valenti, Davide Bacciu, Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, Alexander Lerchner, Arka Higgins, Loic Pal, Nick Matthey, Guillaume Watters, Alexander Desjardins, Hyunjik Kim, Andriy Mnih, Xuechen Ricky Tq Chen, Roger Li, David Grosse, Duvenaud, Shengjia Zhao, Jiaming Song, Stefano Ermon, Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schoelkopf, Olivier Bachem, Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, Marc'aurelio Ranzato, Ben Poole, Gunnar Rtsch, Bernhard Schlkopf, Michael Tschannen, Junxiang Chen, Kayhan Batmanghelich, Junwen Bai, Weiran Wang, Carla Gomes, Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, Abhishek Kumar, Prasanna Sattigeri, Avinash Balakrishnan, Cian Eastwood, K Christopher, WilliamsOpenalex Unknown 2022 · DOI · OpenAlex

Abstract

The recently introduced weakly disentangled representations proposed to relax some constraints of the previous definitions of disentanglement, in exchange for more flexibility. However, at the moment, weak disentanglement can only be achieved by increasing the amount of supervision as the number of factors of variations of the data increase. In this paper, we introduce modular representations for weak disentanglement, a novel method that allows to keep the amount of supervised information constant with respect the number of generative factors. The experiments shows that models using modular representations can increase their performance with respect to previous work without the need of additional supervision.

Introduction

In representation learning, disentanglement aims to encode data such that distinct components of the latent code correspond to specific physical factors, such as color, shape, or position. While traditional "strict" disentanglement (mapping one dimension to one factor) is often too restrictive for complex real-world data, weak disentanglement offers more flexibility by encoding factors across regions of the latent space.

However, existing weak disentanglement methods suffer from a significant scaling bottleneck: the required amount of supervised "hint" data grows rapidly as the number of generative factors increases. This paper introduces a modular approach that achieves high-quality weak disentanglement and controllable data manipulation while keeping the supervised information constant, regardless of data complexity.

Methodology: The Modular AutoEncoder (M-AE)

The proposed Modular AutoEncoder (M-AE) architecture addresses the supervision bottleneck through four key pillars:

  1. Partitioned Latent Space: The encoder partitions the latent space into $G$ separate codes, where $G$ corresponds to the number of generative factors (e.g., separate partitions for "shape," "position," and "rotation").
  2. Adaptive Modular Priors: Each partition is assigned a mixture-of-Gaussians prior. The parameters are estimated using a small, fixed-size subset of annotated samples. Because partitions are independent, labels can be reused across all factors simultaneously.
  3. Adversarial Training: A discriminator network ensures the latent codes produced by the encoder match the structured, modular priors, forcing the model to respect the partitioned organization.
  4. Relational Learner (ReL): This sub-network learns to perform controlled manipulations, such as altering a "shape" code while preserving "position" and "color" information.
Architecture of the Modular AutoEncoder (M-AE) featuring partitioned latent codes and a Relational Learner (ReL).

Experimental Evaluation

The model was evaluated on standard benchmarks: dSprites (2D shapes with varying scale, rotation, and position) and Shapes3D (3D objects with varying colors, shapes, and orientations). To test scalability, multiple versions of these datasets (v2 to v5) were used, featuring increasing numbers of factor combinations.

Key Insights

  • Supervision Efficiency: Modularizing the latent space allows the model to generalize across factor combinations. Understanding "red" and "square" independently enables the model to represent a "red square" without specific joint supervision.
  • Independence as a Constraint: Applying KL-divergence loss to individual partitions ($z_i$) rather than the entire vector ($z$) encourages the separation of information.
  • Robustness of Weak Disentanglement: The results suggest that using regions (weak disentanglement) is more robust than single dimensions (strict disentanglement) when supported by a modular architecture.

Results

The modular approach significantly outperforms previous state-of-the-art methods in both accuracy and label efficiency.

Comparison of accuracy and supervision requirements (represented by $\tau$) against previous work across different dataset complexities.

As shown in the results, for the most complex Shapes3D task (v5), previous work required 12,000 supervised samples to achieve an accuracy of 0.124. In contrast, the M-AE achieved a much higher accuracy of 0.633 using only 1,000 samples. Furthermore, the model achieved nearly perfect factor separation on dSprites, with a DCI (Disentanglement, Completeness, Informativeness) score of 0.9732, indicating superior performance across standard metrics.

Conclusion and Future Work

Modular representations provide a robust framework for weak disentanglement, offering superior scalability and controllability compared to non-modular methods. While highly effective for categorical factors, future research will focus on extending this approach to continuous factors, such as smooth lighting gradients. Additionally, reducing the reliance on prior knowledge regarding the number of generative factors remains an open challenge for fully unsupervised discovery.

Summary updated

Abstract

The recently introduced weakly disentangled representations proposed to relax some constraints of the previous definitions of disentanglement, in exchange for more flexibility. However, at the moment, weak disentanglement can only be achieved by increasing the amount of supervision as the number of factors of variations of the data increase. In this paper, we introduce modular representations for weak disentanglement, a novel method that allows to keep the amount of supervised information constant with respect the number of generative factors. The experiments shows that models using modular representations can increase their performance with respect to previous work without the need of additional supervision.
BibTeX
@inproceedings{valenti2022esann,
  author = {Andrea Valenti and Davide Bacciu and Andrea Valenti and Davide Bacciu and Irina Higgins and Loic Matthey and Arka Pal and Christopher Burgess and Xavier Glorot and Matthew Botvinick and Shakir Mohamed and Alexander Lerchner and Irina Christopher P Burgess and Arka Higgins and Loic Pal and Nick Matthey and Guillaume Watters and Alexander Desjardins and Lerchner and Hyunjik Kim and Andriy Mnih and Xuechen Ricky Tq Chen and Roger Li and David Grosse and Duvenaud and Shengjia Zhao and Jiaming Song and Stefano Ermon and Francesco Locatello and Stefan Bauer and Mario Lucic and Gunnar Raetsch and Sylvain Gelly and Bernhard Schoelkopf and Olivier Bachem and Guillaume Lample and Neil Zeghidour and Nicolas Usunier and Antoine Bordes and Ludovic Denoyer and Marc'aurelio Ranzato and Francesco Locatello and Ben Poole and Gunnar Rtsch and Bernhard Schlkopf and Olivier Bachem and Michael Tschannen and Junxiang Chen and Kayhan Batmanghelich and Junwen Bai and Weiran Wang and Carla Gomes and Ian Goodfellow and Jean Pouget-Abadie and Mehdi Mirza and Bing Xu and David Warde-Farley and Sherjil Ozair and Aaron Courville and Yoshua Bengio and Abhishek Kumar and Prasanna Sattigeri and Avinash Balakrishnan and Cian Eastwood and K Christopher and Williams},
  title = {{ESANN 2022 proceedings}},
  booktitle = {Openalex Unknown},
  year = {2022},
  doi = {10.14428/esann/2022},
}
Figure