Improving Deep Neural Networks with Probabilistic Maxout Units

Jost Tobias Springenberg, Martin Riedmiller

Abstract

"Probout" (Probabilistic Maxout) is a stochastic activation function that replaces the "winner-takes-all" nature of standard Maxout units with a sampling procedure. By allowing gradients to flow through multiple linear sub-units during training, Probout improves a neural network's ability to learn invariant features, ultimately achieving state-of-the-art performance on standard image classification benchmarks.

The Problem with Standard Maxout

The Maxout unit is a popular activation function designed to synergize well with Dropout. A Maxout unit applies several linear transformations to an input and outputs only the maximum value among them—a process known as "subspace pooling."

However, this approach has two primary limitations:

  1. Gradient Sparsity: Because Maxout only passes forward the maximum value, the gradient only flows through one "winning" sub-unit during backpropagation. The other $k-1$ sub-units receive no learning signal, potentially leaving their representational capacity underutilized.
  2. Invariance Limits: While Maxout provides some degree of invariance to input changes, a more flexible, probabilistic approach could help the network learn even more robust features, such as invariance to small rotations or translations.

Methods: Introducing Probabilistic Maxout (Probout)

To address these limitations, the authors propose Probabilistic Maxout (Probout). Instead of strictly picking the maximum value $z_i$ from a set of $k$ linear transformations, Probout treats the activations as a probability distribution.

Comparison of pooling strategies: spatial pooling, maxout, and probout.
  • The Sampling Mechanism: Probout uses a Boltzmann distribution (similar to a Softmax function) to assign a probability to each sub-unit based on its activation value. A hyperparameter $\lambda$ (inverse temperature) controls how "peaky" or uniform this distribution is.
  • Training: During the forward pass, one sub-unit is sampled based on these probabilities. Over many training iterations, all sub-units that produce reasonably high activations get a chance to "see" the gradient and update their weights.
  • Integration with Dropout: The authors elegantly combine Dropout directly into the sampling step. By adding a "zero-activation" state to the probability distribution, they can effectively drop the entire unit with a 50% chance, streamlining the regularization process.
  • Inference: At test time, because the sampling is stochastic, the model averages the results of multiple forward passes (typically 50) to obtain a stable, robust prediction.

Key Insights and Feature Learning

  • Better Subspace Utilization: By sampling, the network forces all $k$ sub-units to learn useful features. In standard Maxout, if one unit starts "winning" early in training, the others might never catch up.
  • Learning Invariance: Visualizations of the learned filters demonstrate that Probout sub-units often learn "transformed" versions of the same feature (e.g., the same edge at slightly different rotations). This confirms that the probabilistic approach successfully encourages the network to learn internal invariances.
  • The $\lambda$ Schedule: The authors found that "annealing" $\lambda$—starting with a low value to encourage exploration across sub-units and increasing it over time—yielded the best performance.
  • Layer-Specific Needs: Interestingly, probabilistic sampling is most beneficial in the lower layers of a network (where learning spatial invariance is critical). Conversely, the higher layers perform better when they behave more like standard Maxout, which facilitates model averaging.

Experimental Results

The model was evaluated on three classic image classification benchmarks, matching or exceeding the state-of-the-art (SOTA) at the time of publication:

  • CIFAR-10 (60,000 32x32 images, 10 classes): Achieved an error rate of 11.35% (without data augmentation) and 9.39% (with augmentation), slightly outperforming the original Maxout baseline.
  • CIFAR-100 (100 classes): Achieved 38.14% error, a notable improvement over the Maxout baseline of 38.57%.
  • SVHN (Street View House Numbers): Achieved 2.39% error, setting a new SOTA for models trained without heavy data augmentation.

Limitations and Future Work

The primary drawback of Probout is its test-time complexity. Because it requires multiple forward passes (sampling) at test time to achieve peak accuracy, it is computationally more expensive during deployment than standard Maxout or ReLU networks. Furthermore, performance is sensitive to the choice and scheduling of the $\lambda$ hyperparameter, adding complexity to the training process.

The authors note that developing a more efficient "approximate inference" scheme—similar to how standard Dropout uses weight scaling at test time—would be a highly valuable direction for future work to bypass the cost of multiple forward passes.

Summary updated

Abstract

We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).
BibTeX
@inproceedings{springenberg2014improving,
  author = {Jost Tobias Springenberg and Martin Riedmiller},
  title = {{Improving Deep Neural Networks with Probabilistic Maxout Units}},
  booktitle = {International Conference on Learning Representations},
  year = {2014},
  url = {https://openreview.net/forum?id=NR4KjDE0w9RXD},
}
Figure