Efficient Dynamic Batch Adaptation (Student Abstract)

Cristian Simionescu, George StoicaAAAI 2023 · DOI

Abstract

Efficient Dynamic Batch Adaptation (EDBA) is a novel training methodology designed to enable deep learning models to perform effectively under extreme data scarcity. By leveraging last-layer gradients as a proxy for sample utility and employing a randomized subset selection strategy, EDBA dynamically optimizes batch composition and size. Experimental results on CIFAR-10 and CIFAR-100 demonstrate that EDBA significantly outperforms standard SGD baselines, achieving over 90% accuracy on CIFAR-10 with as few as 100 samples, while also halving training time on full-scale datasets.

Introduction

The success of deep learning typically hinges on the availability of massive labeled datasets. However, in specialized fields such as rare disease diagnostics or niche industrial applications, data is often a scarce resource. While Dynamic Batch Adaptation (DBA) was previously proposed to select optimal training samples, its computational complexity—requiring per-sample gradients for every layer—rendered it impractical for modern deep architectures like ResNets. This work introduces Efficient Dynamic Batch Adaptation (EDBA), a scalable approach that maintains the benefits of dynamic selection without the prohibitive overhead.

Methods

EDBA optimizes the training process by dynamically adjusting the composition and size of each batch. Two primary innovations facilitate this efficiency:

  1. Last-Layer Proxy: To avoid the cost of full-network gradient computation, EDBA calculates per-sample gradients only for the final layer after a forward pass. This proxy effectively identifies beneficial samples for the current update step.
  2. Randomized Subset Selection: Instead of exhaustive search, EDBA evaluates several random subsets of a candidate batch. It selects the subset that minimizes a margin-based loss, which balances the model's current error against gradient magnitude. This allows the model to prioritize high-signal samples early in training and filter noise during convergence.

The algorithm dynamically scales the batch size (ranging from 10 to 1000), increasing it in later stages to average out noise and improve generalization.

Key Results

The efficacy of EDBA was evaluated using PreResNet56 on CIFAR-10 and CIFAR-100 under various data constraints.

Test accuracy comparison between Baseline and EDBA across varying data budgets on CIFAR-10 and CIFAR-100.

As shown in the table above, EDBA's performance gains are most prominent in data-poor scenarios:

  • Extreme Scarcity: With only 100 samples of CIFAR-10, EDBA achieved 90.68% accuracy, whereas the baseline struggled at 23.79%.
  • 1% Data Regime: On CIFAR-100 with 1% of the data, EDBA reached 68.85% accuracy, a massive improvement over the baseline's 8.40%.
  • Efficiency: On the full CIFAR-10 dataset, EDBA reached convergence in approximately 120 epochs, compared to 300 epochs for the baseline, effectively cutting training time by more than half.

Discussion and Limitations

EDBA acts as a force multiplier for small datasets by intelligently filtering noise and focusing on high-utility samples. However, several avenues for future research remain. The current evaluation assumes clean datasets; future work should investigate EDBA's robustness against mislabeled or "poisoned" data. Additionally, while highly effective for image classification, the method's applicability to Natural Language Processing (NLP) and other modalities requires further validation. Finally, more extensive wall-clock time comparisons against modern optimizers like Adam or SAM would provide a more comprehensive view of its practical efficiency.

Summary updated

BibTeX
@inproceedings{simionescu2023efficient,
  author = {Cristian Simionescu and George Stoica},
  title = {{Efficient Dynamic Batch Adaptation (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27024},
}
Figure