Natural representation of composite data with replicated autoencoders

Matteo Negri, Davide Bergamini, Carlo Baldassi, Riccardo Zecchina, Christoph FeinauerarXiv 2019 · arXiv

Introduction

In fields such as genomics and natural language processing, data is inherently "composite"—a single observation, like a protein sequence, represents a mixture of various underlying features such as evolutionary patterns or functional domains. Identifying these basic building blocks from noisy, unlabeled datasets is a significant challenge due to highly non-linear interactions. While standard tools like sparse autoencoders attempt to find these features by compressing data into a bottleneck layer, they often fall into "sharp minima" in the loss landscape. This leads to overfitting on training noise rather than learning the true generative structure of the data.

To address this, Negri et al. (2019) introduced Replicated Autoencoders (R-AE), an unsupervised learning method that shifts the optimization focus from standard loss to "local entropy." By targeting robust, flat regions of the loss landscape, the model discovers more biologically meaningful representations that traditional neural networks frequently miss.

Methods: Replicated Autoencoders and Local Entropy

The R-AE architecture utilizes a feed-forward autoencoder with a small bottleneck hidden layer, employing $L_1$ regularization on activations to enforce sparsity. The core innovation lies in the training protocol inspired by statistical physics.

Instead of optimizing a single model, the researchers train a system of $R$ identical "replicas" coupled to a central configuration. This setup optimizes for local entropy, searching for "wide" or "flat" minima rather than the absolute lowest point (which often represents a narrow, overfitted solution). These flat regions are more robust to perturbations and are significantly more likely to represent the true underlying generative process of the data.

Architecture of the Replicated Autoencoder (A) and the conceptual loss landscape (B) showing the search for flat minima.

Datasets and Experimental Setup

The approach was validated using two primary data sources:

  1. Synthetic Data: Datasets where "ground truth" features were known, created by combining binary sparse vectors via a non-linear "min" function to simulate complex feature interactions.
  2. Biological Data: 18 protein families from the PFAM database (e.g., ribosomal domains), processed as one-hot encoded amino acid sequences.

Key Insights and Results

A critical discovery was the identification of a "knee point" in the loss curve relative to the sparsity regularizer ($\gamma$). This point marks a phase transition where the model's internal representation becomes most highly correlated with the true underlying features.

Feature Retrieval Performance

On synthetic data, the R-AE demonstrated substantial improvements over standard autoencoders (S-AE). For a dictionary size of 160, the R-AE retrieved 100% of the basic features, while the S-AE failed significantly. As the dictionary size $D$ increased to 240 and beyond, the R-AE continued to outperform the standard model, identifying approximately 40% more features.

Performance comparison between S-AE and R-AE: (A) Feature retrieval and test loss vs. regularizer strength $\gamma$, highlighting the "knee point"; (B) Retrieval success across different dictionary sizes $D$.

Biological Relevance and Clustering

When applied to protein sequences, the hidden units of the R-AE showed a significantly higher correlation with taxonomic labels than standard models. This indicates that the R-AE successfully "discovered" evolutionary signatures without explicit supervision. Furthermore, the latent representations learned by the R-AE improved the accuracy of downstream clustering algorithms compared to using raw input data or standard autoencoder outputs.

Conclusion and Future Directions

The R-AE framework demonstrates that the geometry of the loss landscape is as vital as the loss value itself for generalization. While performance challenges remain as dictionary sizes grow extremely large, the method offers a robust path forward for unsupervised feature extraction. The authors suggest that this approach could be particularly valuable for single-cell transcriptomics, where identifying co-regulated gene expression modules remains a critical hurdle.

Summary updated

Abstract

Generative processes in biology and other fields often produce data that can be regarded as resulting from a composition of basic features. Here we present an unsupervised method based on autoencoders for inferring these basic features of data. The main novelty in our approach is that the training is based on the optimization of the `local entropy' rather than the standard loss, resulting in a more robust inference, and enhancing the performance on this type of data considerably. Algorithmically, this is realized by training an interacting system of replicated autoencoders. We apply this method to synthetic and protein sequence data, and show that it is able to infer a hidden representation that correlates well with the underlying generative process, without requiring any prior knowledge.
BibTeX
@misc{negri2019natural,
  author = {Matteo Negri and Davide Bergamini and Carlo Baldassi and Riccardo Zecchina and Christoph Feinauer},
  title = {{Natural representation of composite data with replicated autoencoders}},
  year = {2019},
  eprint = {1909.13327},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.13327},
}
Figure