Evolutionary Approach for AutoAugment Using the Thermodynamical Genetic Algorithm

Akira Terauchi, Naoki MoriAAAI 2021 · DOI

Abstract

Data augmentation is a critical component in training robust deep learning models, yet the automated discovery of optimal augmentation policies remains computationally expensive and prone to premature convergence. This paper introduces TDGA AutoAugment, a novel approach that utilizes the Thermodynamical Genetic Algorithm (TDGA) to balance model accuracy (energy) with population diversity (entropy). By explicitly managing this trade-off, the method achieves state-of-the-art results on benchmark datasets like CIFAR-10 and SVHN in just 2.5 GPU hours, representing a significant efficiency gain over the 5,000 hours required by the original AutoAugment.

Introduction

While automated tools like AutoAugment (AA) have replaced manual tuning of data augmentation policies, they face two primary challenges: extreme computational costs and a lack of diversity in the discovered policies. Search algorithms often converge on a single dominant transformation, such as rotation, which can limit a model's ability to generalize across varied real-world data. The authors propose a search controller that is both fast and capable of maintaining a diverse set of augmentation sub-policies through a thermodynamically inspired evolutionary process.

Methods: The Thermodynamical Genetic Algorithm

The core innovation lies in the application of the Free Energy Principle to policy search. In TDGA, individuals for the next generation are selected based on their "Free Energy" ($F$), defined as:

$$F = E - HT$$

  • $E$ (Energy): Corresponds to the model's error (negative validation accuracy). Minimizing energy ensures the policy is effective.
  • $H$ (Entropy): Measures the diversity of the genotypes (transformations) within the population.
  • $T$ (Temperature): A hyperparameter controlling the trade-off between raw accuracy and diversity.
The TDGA evolution loop for discovering augmentation policies.

To further improve efficiency, the search space is reduced from $10^{32}$ to approximately $10^8$ by sharing magnitude and probability across operations, similar to the RandAugment approach. Additionally, sub-policies are evaluated using a pre-trained model on a validation set, avoiding the need to train from scratch at every step.

Key Insights and the Temperature Parameter

A central finding of the research is the role of the temperature parameter $T$. It acts as a bridge between different augmentation philosophies. At low temperatures, the system behaves like a greedy search focused solely on accuracy. At high temperatures, it mirrors RandAugment, where transformations are chosen with high randomness. The "sweet spot" allows the algorithm to discover a curated, diverse set of operations that outperform purely random or purely greedy methods.

Test accuracy as a function of TDGA temperature, showing the trade-off between accuracy and diversity.

By treating diversity as a first-class citizen through the entropy term ($H$), TDGA prevents the population from becoming a monoculture. The algorithm successfully identifies and phases out harmful operations (e.g., Solarize on specific tasks) while retaining beneficial ones like Rotate and Translate.

Experimental Results

The authors evaluated TDGA AutoAugment on CIFAR-10 and SVHN using Wide-ResNet architectures.

  • Efficiency: The search required only 2.5 GPU hours on an RTX 2080Ti, a massive reduction compared to the 5,000 hours for the original AA and 5 hours for Population Based Augmentation (PBA).
  • Accuracy: On CIFAR-10 (Wide-ResNet-28-2), the method achieved 95.92% accuracy, slightly exceeding both the original AutoAugment (95.9%) and RandAugment (95.8%). On SVHN, it reached 97.95%, remaining competitive with state-of-the-art automated methods.

Conclusion and Future Work

TDGA AutoAugment demonstrates that explicitly balancing diversity and accuracy can lead to highly efficient and effective data augmentation policies. Future research may explore dynamic temperature schedules—similar to simulated annealing—to allow for greater exploration in early generations and refinement in later ones. Additionally, optimizing the order of operations using techniques like the Traveling Salesman Problem (TSP) could further enhance policy performance.

Summary updated

BibTeX
@inproceedings{terauchi2021evolutionary,
  author = {Akira Terauchi and Naoki Mori},
  title = {{Evolutionary Approach for AutoAugment Using the Thermodynamical Genetic Algorithm}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i11.17184},
}
Figure