Post-training Iterative Hierarchical Data Augmentation for Deep Networks

Adil Khan, Khadija FrazNeurIPS 2020 · OpenReview

Introduction

Deep neural networks are notorious for overfitting to training data, leading to poor generalization on unseen test sets. While Data Augmentation (DA)—like flipping or rotating images—is a standard fix, it has two major drawbacks:

  1. Domain Specificity: Traditional DA is often designed for images and doesn't translate easily to other data types, such as wearable sensor signals.
  2. Input Space Limitations: Performing DA only on the raw input ignores the rich, hierarchical representations a network learns in its deeper layers.

Iterative Hierarchical Data Augmentation (IHDA) is a post-training technique designed to overcome these limitations. By synthesizing new samples directly within the network's learned feature spaces, IHDA boosts generalization. It identifies "hard-to-learn" regions and uses Variational Autoencoders (VAEs) to generate targeted data points layer-by-layer, achieving state-of-the-art performance across image classification and sensor-based activity recognition tasks.

Methodology

Existing feature-space augmentation methods often apply simple transformations or augment the entire dataset blindly. In contrast, IHDA focuses on the "difficult" parts of the feature manifold—regions where classes overlap and the model is most likely to make mistakes. The approach is a multi-step, post-training refinement process:

  1. Iterative Refinement: Unlike standard training, IHDA starts with a pre-trained model and moves through the network layer-by-layer (or block-by-block).
  2. Targeting "Hard" Samples: For a given layer, the algorithm analyzes the neighborhood of every data point in the feature space using Radial Basis Functions (RBFs). If a point is surrounded by many samples of different classes, it’s marked as "hard-to-classify" and selected for augmentation.
  3. VAE-based Synthesis: A VAE is trained on the feature activations of the current layer. For the identified "hard" points, the VAE generates new, synthetic feature vectors. A noise parameter ensures semantic variety in these new samples.
  4. Hierarchical Fine-tuning: These synthetic features are fed into the subsequent layers of the network. The weights of all following layers are then fine-tuned to accommodate this more difficult version of the dataset.
  5. Adaptive Dropout: To manage computational costs and prevent overfitting during this second phase, the algorithm uses a "dropout" mechanism to skip certain layers, prioritizing deeper layers which contain more meaningful semantic information.

Key Insights

  • Manifold Unfolding: Deep networks "unfold" data manifolds. By the time data reaches deeper layers, representations are more linear and structured, making synthetic data generated there more credible than data synthesized in the raw input space.
  • Quality Over Quantity: IHDA doesn't just add more data; it adds difficult data. Focusing on the boundaries between classes forces the model to learn more robust decision boundaries.
  • Domain Agnosticism: Because the VAE learns the distribution of abstract features rather than raw pixels, the method works just as well for sensor data as it does for images.

Key Results

The authors evaluated IHDA on several high-profile benchmarks, including CIFAR-10, CIFAR-100, ImageNet, and the SHL (Sussex-Huawei Locomotion-Transportation) dataset for non-image wearable sensor data. IHDA consistently outperformed baseline models and other state-of-the-art augmentation strategies like AutoAugment (AA) and Population Based Augmentation (PBA).

Image Classification Performance: On CIFAR-100, using a Wide-ResNet-28-10 architecture, IHDA reduced the test error from 18.76% to 13.19%.

CIFAR-100 Test Error Rates

Similarly, on the massive ImageNet dataset, a ResNet-200 model saw its Top-1 accuracy improve from 78.5% to 82.9%.

ImageNet Accuracy

Sensor Data and Efficiency: The most dramatic improvement was seen on the SHL dataset, where recognition accuracy jumped from 83% to 92%, proving the method's effectiveness outside of computer vision. Furthermore, despite being an iterative process, IHDA only adds about 30% to the total training time, which is significantly faster than training some meta-learning-based augmentation policies from scratch.

Limitations and Future Work

  • Hyperparameter Tuning: The method relies on sensitive hyperparameters, such as the neighborhood distance and the dropout probability, which require a validation set to optimize for each specific architecture.
  • Computational Overhead: While a 30% increase in training time is reasonable for many applications, it may still pose a barrier for researchers with limited compute resources or extremely large-scale models.
  • Integration: Currently presented as a post-training step, future work could explore integrating this hierarchical approach directly into the initial training loop for potentially better results or faster convergence.

Summary updated

Abstract

In this paper, we propose a new iterative hierarchical data augmentation (IHDA) method to fine-tune trained deep neural networks to improve their generalization performance. The IHDA is motivated by three key insights: (1) Deep networks (DNs) are good at learning multi-level representations from data. (2) Performing data augmentation (DA) in the learned feature spaces of DNs can significantly improve their performance. (3) Implementing DA in hard-to-learn regions of a feature space can effectively augment the dataset to improve generalization. Accordingly, the IHDA performs DA in a deep feature space, at level l, by transforming it into a distribution space and synthesizing new samples using the learned distributions for data points that lie in hard-to-classify regions, which is estimated by analyzing the neighborhood characteristics of each data point. The synthesized samples are used to fine-tune the parameters of the subsequent layers. The same procedure is then repeated for the feature space at level l+1. To avoid overfitting, the concept of dropout probability is employed, which is gradually relaxed as the IHDA works towards high-level feature spaces. IHDA provided a state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet for several DNs, and beat the performance of existing state-of-the-art DA approaches for the same networks on these datasets. Finally, to demonstrate its domain-agnostic properties, we show the significant improvements that IHDA provided for a deep neural network on a non-image wearable sensor-based activity recognition benchmark.
BibTeX
@inproceedings{khan2020post,
  author = {Adil Khan and Khadija Fraz},
  title = {{Post-training Iterative Hierarchical Data Augmentation for Deep Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=6iwVAJ1_vLC},
}
Figure