Meta Dropout: Learning to Perturb Latent Features for Generalization

Haebeom Lee, Taewook Nam, Eunho Yang, Sung Ju HwangICLR 2020 · OpenReview

TL;DR

Meta Dropout is a novel regularization technique that replaces "blind" random noise with a meta-learned, input-dependent noise generator. By explicitly training the model to perturb its own training features to minimize error on a held-out test set, it achieves superior generalization and surprising adversarial robustness in few-shot learning tasks.

What Problem Does This Paper Address?

The "Holy Grail" of machine learning is generalization: the ability of a model to perform well on data it has never seen before. Traditionally, we use regularization techniques like Dropout (randomly zeroing out neurons) or Mixup (interpolating between training examples) to prevent the model from memorizing the training set.

However, these methods are largely heuristic. They apply noise "blindly" without knowing exactly how the training data should be perturbed to better represent the unknown test distribution. In a standard learning framework, this is an impossible problem because the test data is unobservable during training. The authors ask: Can we learn a noise generator that specifically targets better generalization?

Conceptual illustration of how perturbing training data helps generalization

Methods

The authors leverage the meta-learning (or "learning to learn") framework to solve this. In meta-learning, a model is trained on a series of "episodes," each containing a small training set (support set) and a test set (query set).

The Core Innovation: Meta Dropout Instead of using a fixed dropout rate, the authors introduce a noise generator network.

  1. Input-Dependent Perturbation: For every training instance, the noise generator looks at the latent features and outputs a specific multiplicative noise distribution.
  2. The Meta-Objective: The noise generator is optimized so that when its noise is applied to the training (support) data, the resulting model achieves the lowest possible loss on the test (query) data.
  3. Stochasticity: They use a "Softplus" transformation of a Gaussian distribution to ensure the noise is positive and differentiable, allowing them to use the reparameterization trick for end-to-end training.

Essentially, the model learns to "hallucinate" the most useful variations of its training data to prepare itself for the challenges of the test set.

Datasets

The researchers evaluated Meta Dropout on two gold-standard benchmarks for few-shot classification:

  • Omniglot: A "transpose" of MNIST containing 1,623 different handwritten characters from 50 alphabets. It is used for 20-way classification (choosing the right character out of 20 options).
  • miniImageNet: A more complex subset of the famous ImageNet dataset, consisting of 100 classes with 600 images each, resized to 84x84. It is used for 5-way classification.

Key Insights and Ideas

  • Regularization as Meta-Knowledge: The noise generator captures "meta-knowledge" about how features typically vary across different tasks. This knowledge is transferable to entirely new, unseen tasks at test time.
  • Input-Dependence is Crucial: The paper shows that "one-size-fits-all" noise is sub-optimal. By tailoring the perturbation to the specific input (e.g., perturbing different features for a "dog" vs. a "car"), the model learns a much more robust decision boundary.
  • Generalization vs. Robustness: One of the most striking findings is that Meta Dropout significantly improves adversarial robustness. Even though the model wasn't explicitly trained to fight off hackers, the act of learning to perturb features for generalization naturally made it resistant to PGD (Projected Gradient Descent) attacks.
  • Bridge to Variational Inference: The authors provide a theoretical bridge, showing that their lower-bound objective can be interpreted as performing variational inference on a specific graphical model where latent variables represent the perturbations.

Key Results

The authors integrated Meta Dropout into popular meta-learning baselines like MAML and Meta-SGD.

Results table and decision boundary visualizations comparing MAML and Meta-dropout
  • Performance Boost: On miniImageNet (5-way, 1-shot), Meta Dropout improved MAML's accuracy from 49.58% to 51.93%. On Omniglot (20-way, 1-shot), it pushed Meta-SGD from 96.16% to 97.02%.
  • Outperforming Heuristics: Meta Dropout consistently outperformed more traditional regularizers like Manifold Mixup, Information Bottleneck, and Information Dropout.
  • Adversarial Edge: In tests against adversarial attacks, Meta Dropout maintained much higher accuracy than the base models, which often crashed to near-zero accuracy under the same conditions.

Limitations and Future Work

  • Computational Complexity: Like many meta-learning methods (especially those based on MAML), Meta Dropout requires computing second-order derivatives, which can be memory-intensive and slow for very large networks.
  • Task-Specific Structure: The method relies on the episodic structure of meta-learning. Applying this to "standard" supervised learning (where there aren't clear task splits) would require a different formulation.
  • Future Directions: The authors suggest further investigating the link between generalization and adversarial robustness. They also note that while multiplicative noise worked best here, other forms of noise (like additive noise) might be better suited for different types of data or architectures.

Summary updated

BibTeX
@inproceedings{lee2020meta,
  author = {Haebeom Lee and Taewook Nam and Eunho Yang and Sung Ju Hwang},
  title = {{Meta Dropout: Learning to Perturb Latent Features for Generalization}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=kxIzoki5mV},
}
Figure