Introduction
In representation learning, "disentanglement" refers to the ability of a model to isolate independent generative factors of an image—such as color, shape, and position—into distinct components of its latent space. Traditional approaches, like $\beta$-VAEs, typically rely on localist representations, where individual coordinates in a latent vector are mapped to specific properties. However, these representations often lack robustness and become increasingly difficult to interpret as dimensionality grows.
ArSyD (Architecture for Symbolic Disentanglement) introduces a shift toward distributed representations by leveraging Hyperdimensional Computing (HDC). Instead of single numerical values, ArSyD represents generative factors as high-dimensional "symbolic" hypervectors, enabling more robust, noise-resistant, and interpretable image manipulation through algebraic operations.
Methodology: Symbolic Disentanglement
The core innovation of ArSyD lies in its use of Vector Symbolic Architectures (VSA). In this framework, properties are not just indices but full-sized vectors that can be combined via superposition.
The ArSyD Architecture
The model consists of an encoder that extracts features from an input image, followed by a Generative Factor (GF) Projection module. This module uses an attention mechanism to select property values from a fixed codebook (item memory). These values are then combined using HDC operations:
- Bundling (Addition): Acts like a set operation, allowing multiple independent properties to coexist in the same vector space.
- Binding (Circular Convolution): Links specific attributes to their values (e.g., associating the attribute "Color" with the value "Red").
Weakly Supervised Training
ArSyD is trained using a weakly supervised "feature exchange" paradigm. The model is presented with pairs of images that differ by exactly one factor (e.g., two scenes identical except for the object's color). By swapping the symbolic vectors corresponding to the differing factor between the two images, the model learns to isolate and reconstruct properties accurately.
Evaluation and Metrics
Because standard disentanglement metrics are designed for coordinate-based vectors, the authors introduced two new metrics: Disentanglement Modularity Metric (DMM) and Disentanglement Compactness Metric (DCM). These metrics evaluate disentanglement by observing how changes in the latent space affect a classifier's prediction in the reconstructed pixel space, making them architecture-agnostic.
Key Results
ArSyD was evaluated on benchmarks including dSprites, CLEVR, and CelebA. Key findings include:
- High-Quality Reconstruction: On the CLEVR1 dataset, ArSyD achieved an FID score of 93.72, significantly outperforming FactorVAE (115.61) and $\beta$-VAE (129.68).
- Precise Segmentation: The model reached a 98.37% Intersection over Union (IoU) on dSprites, surpassing standard VAE baselines.
- Multi-Object Compositionality: When combined with Slot Attention, ArSyD successfully modified specific objects within complex scenes (CLEVR5) without affecting the rest of the image, demonstrating the power of symbolic superposition.
Conclusion and Future Work
ArSyD demonstrates that symbolic hypervectors provide a powerful alternative to traditional latent representations. By treating properties as symbols that can be added or subtracted, the model achieves superior interpretability and editing capabilities. Future research aims to reduce the dependency on paired data and scale the architecture to more complex, non-synthetic datasets by integrating pre-trained feature extractors.
Summary updated