Dropout Training for SVMs with Data Augmentation

Ning Chen, Jun Zhu, Jianfei Chen, Ting ChenFrontiers of Computer Science 2015 · DOI · OpenReview

TL;DR

This paper introduces dropout training, a technique previously successful in deep learning for preventing overfitting, to Support Vector Machines (SVMs). They develop novel algorithms that allow dropout to be applied to both linear and non-linear SVMs, significantly improving their accuracy and robustness, especially in challenging scenarios like feature deletion.

What Problem Does This Paper Address?

Overfitting is a common problem in machine learning, where models learn the training data too well and fail to generalize to new, unseen data. Dropout, a method of randomly disabling neurons during training, has proven highly effective in mitigating overfitting in deep neural networks. However, applying dropout to Support Vector Machines (SVMs), a widely used and successful classification algorithm, has been challenging. The core difficulty lies in the non-smooth nature of the SVM's hinge loss function, which makes it hard to analytically compute the expected loss under corrupted data distributions, a requirement for effective dropout training. Existing methods for dropout often rely on approximations or computationally expensive explicit data corruption, which are not ideal for SVMs.

Methods

The authors propose an Iteratively Re-weighted Least Square (IRLS) algorithm as the core solution. This approach leverages data augmentation techniques to derive a variational upper bound for the expected hinge loss under corrupted data. This bound can then be optimized iteratively.

For linear SVMs, they develop an IRLS algorithm that minimizes a re-weighted quadratic loss at each step, with the re-weights analytically updated. This circumvents the non-smoothness of the hinge loss.

For non-linear SVMs with a single layer of latent representations, they extend the data augmentation approach. To handle the non-linearity of the latent features, they employ a first-order Taylor expansion to approximate the objective function, which is then optimized using an iterative algorithm.

They also extend these ideas to logistic regression, providing a unified framework that sheds light on the connections and differences between hinge and logistic losses in the context of dropout training.

Datasets

The paper evaluates their methods on a variety of datasets:

  • Text Classification:
    • Amazon review datasets (books, kitchen, dvd, electronics) for binary classification.
  • Dmoz and Reuters datasets for multi-class classification.
  • Image Classification:
    • CIFAR-10 dataset.
  • Regression:
    • Hotelreview dataset.
  • Robustness Testing:
    • MNIST dataset for the "nightmare at test time" scenario (feature deletion).

Key Insights and Ideas

  • Dropout is effective for SVMs: The paper demonstrates that dropout training can significantly boost the performance of SVMs, a result not widely explored before.
  • IRLS with Data Augmentation as a unifying framework: This approach provides a principled way to handle the expected hinge loss under corrupted data for SVMs, and can be extended to logistic regression.
  • Bridging Hinge and Logistic Loss: The proposed IRLS framework allows for a direct comparison and understanding of how dropout affects different loss functions.
  • Non-linear SVMs benefit greatly: Incorporating latent representations with dropout further enhances performance, especially in image classification.
  • Robustness to Feature Deletion: Dropout-trained SVMs show remarkable resilience to feature deletion at test time, outperforming other methods in challenging "nightmare at test time" scenarios.

Key Results

  • Improved Classification Accuracy: Dropout-trained SVMs consistently outperformed their non-dropout counterparts across various datasets, including text and image classification. For instance, on Amazon datasets, Dropout-SVM and Dropout-LR generally outperformed MCF-Quadratic.
  • Competitive with State-of-the-Art: In many cases, their proposed methods achieved performance comparable to or better than existing state-of-the-art dropout techniques (e.g., MCF-Logistic).
  • Regression Performance Boost: Dropout-SVR showed improved predictive R² scores compared to the MCF-quadratic model for regression tasks.
  • Robustness: In "nightmare at test time" scenarios with significant feature deletion (e.g., >50%), Dropout-SVM demonstrated superior robustness compared to other methods.
  • Efficiency: The proposed IRLS algorithms are computationally efficient, with linear classifiers being comparable in speed to existing MCF methods.

Limitations and Future Work

  • Kernel Trick: The paper notes that incorporating the kernel trick into dropout learning remains an open question.
  • Online Learning: Developing more efficient algorithms for larger datasets, such as online dropout learning, is suggested.
  • Advanced Architectures: Exploring the integration of Dropout-SVM into deep learning architectures or hierarchical Bayes networks is a promising future direction.
  • Informed Dropout Policies: Investigating more sophisticated dropout policies, potentially using reinforcement learning, is also mentioned.
  • Computational Cost for High-Dimensional Data: While efficient for low-dimensional data, high-dimensional cases might still require significant computational resources for matrix inversions, though numerical methods can be employed.

Summary updated

Abstract

Dropout and other feature noising schemes have shown promising results in controlling over-fitting by artificially corrupting the training data. Though extensive theoretical and empirical studies have been performed for generalized linear models, little work has been done for support vector machines (SVMs), one of the most successful approaches for supervised learning. This paper presents dropout training for both linear SVMs and the nonlinear extension with latent representation learning. For linear SVMs, to deal with the intractable expectation of the non-smooth hinge loss under corrupting distributions, we develop an iteratively re-weighted least square (IRLS) algorithm by exploring data augmentation techniques. Our algorithm iteratively minimizes the expectation of a re-weighted least square problem, where the re-weights are analytically updated. For nonlinear latent SVMs, we consider learning one layer of latent representations in SVMs and extend the data augmentation technique in conjunction with first-order Taylor-expansion to deal with the intractable expected non-smooth hinge loss and the nonlinearity of latent representations. Finally, we apply the similar data augmentation ideas to develop a new IRLS algorithm for the expected logistic loss under corrupting distributions, and we further develop a non-linear extension of logistic regression by incorporating one layer of latent representations. Our algorithms offer insights on the connection and difference between the hinge loss and logistic loss in dropout training. Empirical results on several real datasets demonstrate the effectiveness of dropout training on significantly boosting the classification accuracy of both linear and nonlinear SVMs. In addition, the nonlinear SVMs further improve the prediction performance on several image datasets.
BibTeX
@article{chen2015dropout,
  author = {Ning Chen and Jun Zhu and Jianfei Chen and Ting Chen},
  title = {{Dropout Training for SVMs with Data Augmentation}},
  journal = {Frontiers of Computer Science},
  year = {2015},
  doi = {10.1007/s11704-018-7314-7},
}
Figure