Abstract
The recently introduced weakly disentangled representations proposed to relax some constraints of the previous definitions of disentanglement, in exchange for more flexibility. However, at the moment, weak disentanglement can only be achieved by increasing the amount of supervision as the number of factors of variations of the data increase. In this paper, we introduce modular representations for weak disentanglement, a novel method that allows to keep the amount of supervised information constant with respect the number of generative factors. The experiments shows that models using modular representations can increase their performance with respect to previous work without the need of additional supervision.
Introduction
In representation learning, disentanglement aims to encode data such that distinct components of the latent code correspond to specific physical factors, such as color, shape, or position. While traditional "strict" disentanglement (mapping one dimension to one factor) is often too restrictive for complex real-world data, weak disentanglement offers more flexibility by encoding factors across regions of the latent space.
However, existing weak disentanglement methods suffer from a significant scaling bottleneck: the required amount of supervised "hint" data grows rapidly as the number of generative factors increases. This paper introduces a modular approach that achieves high-quality weak disentanglement and controllable data manipulation while keeping the supervised information constant, regardless of data complexity.
Methodology: The Modular AutoEncoder (M-AE)
The proposed Modular AutoEncoder (M-AE) architecture addresses the supervision bottleneck through four key pillars:
- Partitioned Latent Space: The encoder partitions the latent space into $G$ separate codes, where $G$ corresponds to the number of generative factors (e.g., separate partitions for "shape," "position," and "rotation").
- Adaptive Modular Priors: Each partition is assigned a mixture-of-Gaussians prior. The parameters are estimated using a small, fixed-size subset of annotated samples. Because partitions are independent, labels can be reused across all factors simultaneously.
- Adversarial Training: A discriminator network ensures the latent codes produced by the encoder match the structured, modular priors, forcing the model to respect the partitioned organization.
- Relational Learner (ReL): This sub-network learns to perform controlled manipulations, such as altering a "shape" code while preserving "position" and "color" information.
Experimental Evaluation
The model was evaluated on standard benchmarks: dSprites (2D shapes with varying scale, rotation, and position) and Shapes3D (3D objects with varying colors, shapes, and orientations). To test scalability, multiple versions of these datasets (v2 to v5) were used, featuring increasing numbers of factor combinations.
Key Insights
- Supervision Efficiency: Modularizing the latent space allows the model to generalize across factor combinations. Understanding "red" and "square" independently enables the model to represent a "red square" without specific joint supervision.
- Independence as a Constraint: Applying KL-divergence loss to individual partitions ($z_i$) rather than the entire vector ($z$) encourages the separation of information.
- Robustness of Weak Disentanglement: The results suggest that using regions (weak disentanglement) is more robust than single dimensions (strict disentanglement) when supported by a modular architecture.
Results
The modular approach significantly outperforms previous state-of-the-art methods in both accuracy and label efficiency.
As shown in the results, for the most complex Shapes3D task (v5), previous work required 12,000 supervised samples to achieve an accuracy of 0.124. In contrast, the M-AE achieved a much higher accuracy of 0.633 using only 1,000 samples. Furthermore, the model achieved nearly perfect factor separation on dSprites, with a DCI (Disentanglement, Completeness, Informativeness) score of 0.9732, indicating superior performance across standard metrics.
Conclusion and Future Work
Modular representations provide a robust framework for weak disentanglement, offering superior scalability and controllability compared to non-modular methods. While highly effective for categorical factors, future research will focus on extending this approach to continuous factors, such as smooth lighting gradients. Additionally, reducing the reliance on prior knowledge regarding the number of generative factors remains an open challenge for fully unsupervised discovery.
Summary updated