Reinforcement Based Learning on Classification Task Yields Better Generalization and Adversarial Accuracy (Student Abstract)

Shashi Kant GuptaAAAI 2021 · DOI

Abstract

Standard image classification models trained with cross-entropy loss often suffer from overfitting and vulnerability to adversarial attacks. This paper explores an alternative approach: treating classification as a reinforcement learning (RL) task. By optimizing a model using the Vanilla Policy Gradient (VPG) with a simple binary reward signal, the authors demonstrate significantly improved generalization and enhanced robustness against adversarial examples compared to traditional cross-entropy training.

Introduction

Deep neural networks, while powerful, are notoriously "brittle." Tiny, imperceptible perturbations in input images—known as adversarial examples—can easily trick these models into making incorrect predictions. The standard training objective, Cross-Entropy (CE) loss, focuses on minimizing the distance between predicted and true label distributions. However, this focus may contribute to overfitting and poor adversarial robustness. This research investigates whether a reward-based optimization framework, common in Reinforcement Learning, can produce more resilient models with better generalization properties.

Methodology

The classification task is reformulated as a Reinforcement Learning environment where:

  • State ($s_t$): The input image.
  • Action ($a_t$): The predicted class label.
  • Reward ($R_t$): A binary signal ($+1$ for correct, $-1$ for incorrect).
  • Policy: The model’s softmax probability distribution.

Instead of minimizing CE loss, the model is optimized using the Vanilla Policy Gradient (VPG) to maximize expected reward. Unlike CE, which primarily updates based on the correct label's gradient, the RL approach penalizes the specific incorrect class predicted. This "specific penalty" provides a more nuanced feedback signal, potentially helping the network learn more robust decision boundaries.

The "minimal CNN" architecture used to evaluate the RL-based classification approach.

Experimental Setup

The approach was evaluated on the CIFAR-10 dataset using a controlled "minimal CNN" architecture (shown above). This setup allows for a direct comparison of the loss functions' impacts without the confounding variables of massive model architectures.

Key Insights: Generalization and Stability

The experimental results highlight two major advantages of the RL-based approach:

  1. Superior Generalization: The RL-trained model exhibited a remarkably narrow "generalization gap"—the difference between training and test accuracy remained under 2%. In contrast, the CE-trained model showed a significantly larger gap, indicating a higher tendency to overfit.
  2. Adversarial Stability: When subjected to adversarial training using the Fast Gradient Sign Method (FGSM), the CE model's robustness peaked early and then degraded. The RL model, however, showed consistent improvement in adversarial accuracy over time.
Comparison of training, test, and adversarial accuracies over 220 epochs for both CE and RL models. Note the narrower generalization gap and sustained adversarial improvement for RL.

Results

The models were further tested against AutoAttack, a rigorous ensemble of adversarial attacks. While the CE model achieved higher peak "natural" accuracy, its adversarial robustness was significantly lower than that of the RL model by the end of training.

Model Natural Accuracy Adversarial Accuracy (AutoAttack)
RL (220th epoch) 52.85% 30.41%
CE (60th epoch) 50.96% 26.36%
CE (220th epoch) 59.12% 17.95%

Conclusion and Future Work

This study demonstrates that a Reinforcement Learning objective can yield models that generalize better and are more robust to adversarial attacks than those trained with standard cross-entropy. While the current results do not yet reach state-of-the-art levels for CIFAR-10, they provide a promising direction for improving model reliability. Future work will involve scaling this formulation to larger architectures like Wide ResNets and investigating the computational efficiency of RL-based classification on larger datasets.

Summary updated

BibTeX
@inproceedings{gupta2021reinforcement,
  author = {Shashi Kant Gupta},
  title = {{Reinforcement Based Learning on Classification Task Yields Better Generalization and Adversarial Accuracy (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17893},
}
Figure