Rectified Factor Networks

Djork-Arné Clevert, Thomas Unterthiner, Andreas Mayr, Hubert Ramsauer, Sepp Hochreiter

TL;DR

This paper introduces Rectified Factor Networks (RFNs), a novel unsupervised learning method for creating sparse, non-linear, and high-dimensional data representations. RFNs excel at identifying rare events and capturing data covariance, outperforming existing methods like RBMs and autoencoders, and proving effective for pretraining deep networks and discovering biological insights.

What Problem Does This Paper Address?

Deep learning's success hinges on effective input representations, which are often sparse and hierarchical. While existing methods like autoencoders and RBMs can learn representations, they struggle to enforce sparsity and non-negativity simultaneously, and often fail to capture the underlying data covariance structure. This is particularly problematic when dealing with rare or small events in data, such as seldom side effects in drug discovery or subtle genetic variations. The authors aim to develop a method that can efficiently construct representations that are sparse, non-negative, non-linear, use many coding units, and accurately model the data's covariance structure.

Methods

The core innovation is the Rectified Factor Network (RFN), which builds upon factor analysis by incorporating posterior regularization. This technique separates model characteristics from data-dependent constraints, enforcing non-negative and normalized posterior means for the latent variables (codes). The learning algorithm is a generalized alternating minimization (GAM) approach, derived from posterior regularization. Key aspects include:

  • Factor Analysis Model: The underlying generative model is factor analysis, which models data covariance.
  • Posterior Regularization: This method enforces constraints on the model's posterior distribution, ensuring non-negative and normalized latent representations.
  • Generalized Alternating Minimization (GAM): This algorithm uses gradient descent steps for both the E-step (inference) and M-step (parameter update), allowing for efficient computation and GPU implementation. It's designed to converge to a solution that maximizes the RFN objective.
  • Projected Newton/Gradient Methods: The E-step involves projecting the posterior means onto a feasible set (non-negative and normalized), using efficient projected Newton or gradient methods.

Datasets

The paper evaluates RFNs on several benchmark datasets:

  • Vision Datasets: MNIST, CIFAR-10, and CIFAR-100 were used for pretraining deep networks.
  • Gene Expression Data: Data from two pharmaceutical drug discovery studies were used to demonstrate RFNs' ability to find biologically relevant modules.
  • Synthetic Datasets: Nine synthetic datasets (D1-D9) were generated with varying noise levels and bicluster sizes to compare RFNs against other unsupervised methods.

Key Insights and Ideas

  • Enforcing Non-negativity and Normalization: The key insight is that by enforcing non-negative and normalized posterior means, RFNs can achieve both sparsity and prevent "explaining away" of rare signals by noise. This is crucial for identifying subtle but important patterns.
  • GAM for Efficiency: Using a generalized alternating minimization approach with gradient steps allows for efficient learning, especially on large datasets and with many coding units, making it suitable for GPU implementation.
  • RFNs as Powerful Feature Extractors: The results demonstrate that RFNs are not just good at representation learning but also excel as a pretraining technique for deep networks, leading to improved performance.
  • Discovering Rare Biological Events: The application to gene expression data highlights RFNs' ability to uncover small and rare gene modules that were missed by other methods, suggesting significant potential in biological discovery.

Key Results

  • Superior Sparsity and Reconstruction: On benchmark datasets, RFNs achieved significantly sparser codes, lower reconstruction errors, and better covariance structure explanation compared to RBMs, autoencoders, factor analysis, ICA, and PCA. For instance, RFNs yielded the sparsest codes among methods with SP > 10%.
  • Effective Pretraining: When used for pretraining deep networks on vision datasets, RFNs outperformed RBMs and autoencoders, with RFN pretraining leading to the best performance in six out of nine experiments.
  • Biological Discovery: In drug discovery studies, RFNs identified rare and small gene modules that revealed highly relevant biological insights, supporting decision-making in projects.
  • Computational Efficiency: The GAM algorithm and projected Newton/gradient methods enable efficient computation, offering substantial speedups over traditional quadratic programming solvers.

Limitations and Future Work

  • Convergence Proofs: While the paper proves convergence for the GAM algorithm, it notes that dropout regularization, while beneficial, may invalidate these convergence results.
  • Local Maxima: The GAM procedure guarantees convergence to a local maximum of the objective.
  • Future Work: The authors suggest that RFNs have high potential as unsupervised deep learning techniques due to their ability to handle large datasets, sparse coding, and many representational units. The paper also touches upon the possibility of using alternative Gaussian priors to control sparseness.

Summary updated

Abstract

We propose rectified factor networks (RFNs) to efficiently construct very sparse, non-linear, high-dimensional representations of the input. RFN models identify rare and small events in the input, have a low interference between code units, have a small reconstruction error, and explain the data covariance structure. RFN learning is a generalized alternating minimization algorithm derived from the posterior regularization method which enforces non-negative and normalized posterior means. We proof convergence and correctness of the RFN learning algorithm. On benchmarks, RFNs are compared to other unsupervised methods like autoencoders, RBMs, factor analysis, ICA, and PCA. In contrast to previous sparse coding methods, RFNs yield sparser codes, capture the data's covariance structure more precisely, and have a significantly smaller reconstruction error. We test RFNs as pretraining technique for deep networks on different vision datasets, where RFNs were superior to RBMs and autoencoders. On gene expression data from two pharmaceutical drug discovery studies, RFNs detected small and rare gene modules that revealed highly relevant new biological insights which were so far missed by other unsupervised methods.
BibTeX
@inproceedings{clevert2015rectified,
  author = {Djork-Arné Clevert and Thomas Unterthiner and Andreas Mayr and Hubert Ramsauer and Sepp Hochreiter},
  title = {{Rectified Factor Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2015},
  url = {https://openreview.net/forum?id=ByVxyKZ_Wr},
}
Figure