Rectified Factor Networks

Djork-Arné Clevert, Andreas Mayr, Thomas Unterthiner, Sepp Hochreiter

Rectified Factor Networks: A New Approach to Unsupervised Learning

TL;DR

Rectified Factor Networks (RFNs) introduce a novel unsupervised learning approach that combines the statistical rigor of factor analysis with the sparsity and non-negativity of rectified linear units (ReLUs). By employing a unique "posterior regularization" method, RFNs can identify rare, small-scale events in massive datasets—such as specific gene mutations or niche consumer patterns—with significantly higher efficiency and lower error than standard autoencoders or Restricted Boltzmann Machines (RBMs).

The Problem with Current Representations

In deep learning, a model's performance is heavily dependent on the quality of its input representations. Ideally, these representations should be sparse (only a few units active at once) and non-negative (representing the presence of a feature rather than its "degree of absence").

While standard tools like ReLUs and dropout help achieve these properties in supervised learning, existing unsupervised methods like RBMs and Autoencoders often struggle to maintain them while simultaneously capturing the underlying covariance (the relationship structure) of the data. Furthermore, traditional generative models with sparse priors often fail to produce sparse posteriors (the actual codes generated from data). The authors aim to bridge this gap, creating a model capable of handling thousands of code units to find "rare and small events" that typically get lost in the noise of large datasets.

Methodology

RFNs are built on the foundation of Factor Analysis, a classical statistical method that explains data through a set of latent factors. The "Rectified" aspect arises from how the authors constrain the model:

  1. Posterior Regularization: Instead of merely setting a sparse prior, the authors enforce constraints directly on the model’s output (the posterior means). They require these outputs to be non-negative and normalized.
  2. Generalized Alternating Minimization (GAM): To train the model, they utilize a specialized version of the Expectation-Maximization (EM) algorithm.
  3. Fast Optimization: Solving constraints for thousands of units is computationally demanding. The authors developed a gradient projection algorithm (including a projected Newton method) that provides a theoretical speedup of up to $10^{20}$ compared to standard quadratic solvers on datasets like MNIST or CIFAR.
  4. Non-linearity: By enforcing non-negativity, the model naturally introduces non-linearities, allowing these networks to be "stacked" to create deep, hierarchical representations.

Datasets Evaluated

The authors rigorously evaluated RFNs across three distinct domains:

  • Synthetic Biclustering: Nine datasets designed to test if the model can find hidden patterns (biclusters) buried in varying levels of Gaussian noise.
  • Computer Vision Benchmarks: Standard datasets including MNIST (original and with noise/backgrounds), CIFAR-10, NORB, and specialized sets like "convex/concave shapes" and "rectangles."
  • Bioinformatics: Real-world gene expression data from two pharmaceutical drug discovery studies (focused on antipsychotics and oncology).

Key Insights

  • Constraints over Priors: A major insight is that enforcing sparsity and non-negativity via constraints on the posterior is vastly more effective than simply using sparse priors (like Laplacian distributions). Priors do not guarantee the resulting code will be sparse; posterior regularization does.
  • Solving the "Explaining Away" Problem: In models with many units, the system often "explains away" small, real signals as noise. RFNs use normalization constraints to ensure every hidden unit is utilized, forcing the model to pay attention to rare signals.
  • Covariance Matters: Unlike simple autoencoders, RFNs are explicitly designed to model the data's covariance structure, making them better at reconstruction and more robust to noise.

Key Results

  • Sparsity and Accuracy: In head-to-head comparisons, RFNs achieved the highest sparseness (up to 85%) while maintaining the lowest reconstruction error and covariance approximation error compared to RBMs, Denoising Autoencoders (DAE), and PCA.
Comparison of sparsity and error metrics across models
  • Vision Pretraining: When used to pretrain deep neural networks, RFNs outperformed RBMs and Stacked Denoising Autoencoders on 6 out of 9 vision benchmarks. On the "rectangles" dataset, RFN pretraining achieved a test error of 0.63%, compared to 1.99% for the next best method (SDAE).
  • Biological Discovery: In the drug discovery study, RFNs identified a rare "gene module" related to tubulin downregulation. This revealed a genotoxic side effect (the formation of micronuclei) that had been completely missed by other unsupervised methods, providing critical insights for drug safety.
Biological discovery of micronuclei formation identified by RFNs

Limitations and Future Work

While the paper demonstrates impressive results, some limitations remain:

  • Non-convexity: The E-step involves a non-convex optimization problem. While the authors prove convergence for their specific algorithm, non-convex problems can generally be tricky to scale or initialize.
  • Hyperparameters: The model introduces several hyperparameters (like dropout rates and normalization factors) that require careful tuning via a validation set.
  • Stochasticity: While the GAM algorithm allows for stochastic gradients, the full impact of mini-batch training on the stability of the posterior constraints is an area for further exploration.

Ultimately, because RFNs are geared toward large datasets and sparse coding, they exhibit high potential as a foundational tool for unsupervised deep learning in complex, high-dimensional fields like genomics and e-commerce.

Summary updated

Abstract

We propose rectified factor networks (RFNs) to efficiently construct very sparse, non-linear, high-dimensional representations of the input. RFN models identify rare and small events, have a low interference between code units, have a small reconstruction error, and explain the data covariance structure. RFN learning is a generalized alternating minimization algorithm derived from the posterior regularization method which enforces non-negative and normalized posterior means. We proof convergence and correctness of the RFN learning algorithm.On benchmarks, RFNs are compared to other unsupervised methods like autoencoders, RBMs, factor analysis, ICA, and PCA. In contrast to previous sparse coding methods, RFNs yield sparser codes, capture the data's covariance structure more precisely, and have a significantly smaller reconstruction error. We test RFNs as pretraining technique of deep networks on different vision datasets, where RFNs were superior to RBMs and autoencoders. On gene expression data from two pharmaceutical drug discovery studies, RFNs detected small and rare gene modules that revealed highly relevant new biological insights which were so far missed by other unsupervised methods.RFN package for GPU/CPU is available at http://www.bioinf.jku.at/software/rfn.
BibTeX
@inproceedings{clevert2015rectified,
  author = {Djork-Arné Clevert and Andreas Mayr and Thomas Unterthiner and Sepp Hochreiter},
  title = {{Rectified Factor Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2015},
  url = {https://openreview.net/forum?id=ByVxyKZ_Wr},
}
Figure