AutoAssist: A Framework to Accelerate Training of Deep Neural Networks

Jiong Zhang, Hsiang-Fu Yu, Inderjit S. Dhillon

Introduction

Training modern deep neural networks is often a computationally expensive, "brute force" process. Standard Stochastic Gradient Descent (SGD) treats every training example as equally important, sampling them uniformly from the dataset. However, as a model trains, its "return on investment" (ROI) for each data point changes dynamically. Some examples become "easy" and offer negligible gradient updates, while others remain "informative" and crucial for learning.

Spending expensive GPU or TPU cycles calculating gradients for examples the model has already mastered is a massive waste of time and compute. Previous attempts to solve this inefficiency—such as Curriculum Learning or Importance Sampling—often require manual data sorting or introduce so much computational overhead that they actually slow down the total wall-clock training time.

To address this, researchers have proposed AutoAssist, a framework that accelerates deep neural network training by using a lightweight "Assistant" model to filter out uninformative data points dynamically. By offloading this "instance shrinking" to the CPU while the main model trains on the GPU, the system reduces total operations by 30-40% without sacrificing accuracy.

The AutoAssist Framework

The AutoAssist architecture splits the training workload between two distinct models:

  1. The Boss: The main, complex deep neural network being trained (e.g., a ResNet for vision or a Transformer for text).
  2. The Assistant: A very lightweight, "shallow" model (such as logistic regression) that acts as a smart batch generator.

Workflow and Instance Shrinking

The core innovation of AutoAssist is its utility-aware batch generation. The Assistant predicts the "utility" of a data point based on its predicted loss. If the Assistant deems an instance's utility too low—meaning the Boss likely already knows it and won't learn much from it—that instance is "shrunk" or skipped entirely for the current epoch.

Algorithm 1: Utility Aware Batch Generator

Crucially, the Assistant does not stay static. It is trained jointly with the Boss. After the Boss performs a forward pass on a batch, it sends the actual loss values back to the Assistant. The Assistant uses this feedback to continuously update its understanding of what the Boss finds "hard."

To minimize overhead, the Assistant runs asynchronously on the CPU while the Boss runs on the GPU. They communicate via two concurrent queues: a BossQueue for providing filtered training batches and an AssistantQueue for returning loss feedback. This ensures the GPU is never left idling while the Assistant "cleans" the data.

Key Insights

  • Shallow Proxies are Sufficient: One of the most surprising findings is that a simple linear model (the Assistant) is highly capable of predicting the difficulty of instances for a massive, non-linear model (the Boss). You do not need a deep network to identify "easy" data points.
  • ROI-Aware Training: By framing training as a resource allocation problem, the authors demonstrate that skipping 30-40% of the data doesn't just save time—it actually leads to faster convergence because the model concentrates its computational "energy" strictly on the most informative gradients.
  • Overcoming the "Overhead Trap": Many data-filtering papers show theoretical speedups in terms of "number of iterations" but fail to improve "wall-clock time" because the filtering logic itself is too slow. AutoAssist solves this through its asynchronous CPU/GPU design.

Performance and Results

The researchers evaluated AutoAssist across two major domains: Image Classification (using datasets like ImageNet and CIFAR10 with ResNets) and Neural Machine Translation (using the WMT14 dataset with Transformers and LSTMs).

Across these tasks, AutoAssist consistently reached baseline accuracy significantly faster than standard SGD. In NMT tasks, AutoAssist reduced the total operation count (tokens processed) by 30% to 40%.

Convergence comparison of AutoAssist (Shrinking) vs. standard SGD across updates and total time

Furthermore, AutoAssist outperformed other popular acceleration methods like Self-Paced Learning (SPL) and ScreenerNet. Its superiority stems from its ability to adapt to the Boss model dynamically and with vastly lower computational cost, ensuring that reductions in training steps translate directly into real-world time savings.

Limitations and Future Work

While highly effective, AutoAssist does have a few limitations:

  • Assistant Complexity Balance: While linear models worked well for the tasks tested, extremely complex or novel tasks might eventually require slightly more sophisticated Assistants, which could increase CPU overhead and threaten the asynchronous balance.
  • Hyperparameter Sensitivity: The framework relies on a threshold to decide what to skip. While the authors use an exponential moving average to automate this, finding the "sweet spot" for skipping aggressiveness might still require tuning for niche architectures.

Future work could explore extending this "Assistant" approach to other computationally heavy fields, such as Reinforcement Learning or Generative Adversarial Networks (GANs), where data utility is even more volatile and the potential for ROI-aware training is massive.

Summary updated

Abstract

Deep neural networks have yielded superior performance in many contemporary applications. However, the gradient computation in a deep model with millions of instances leads to a lengthy training process even with modern GPU/TPU hardware acceleration. In this paper, we propose AutoAssist, a simple framework to accelerate training of a deep neural network. Typically, as the training procedure evolves, the amount of improvement by a stochastic gradient update varies dynamically with the choice of instances in the mini-batch. In AutoAssist, we utilize this fact and design an instance shrinking operation that is used to filter out instances with relatively low marginal improvement to the current model; thus the computationally intensive gradient computations are performed on informative instances as much as possible. Specifically, we train a very lightweight Assistant model jointly with the original deep network, which we refer to as Boss. The Assistant model is designed to gauge the importance of a given instance with respect to the current Boss such that the shrinking operation can be applied in the batch generator. With careful design, we train the Boss and Assistant in a nonblocking and asynchronous fashion such that overhead is minimal. To demonstrate the effectiveness of AutoAssist, we conduct experiments on two contemporary applications: image classification using ResNets with varied number of layers, and neural machine translation using LSTMs, ConvS2S and Transformer models. For each application, we verify that AutoAssist leads to significant reduction in training time; in particular, 30% to 40% of the total operation count can be reduced which leads to faster convergence and a corresponding decrease in training time.
BibTeX
@inproceedings{zhang2019autoassist,
  author = {Jiong Zhang and Hsiang-Fu Yu and Inderjit S. Dhillon},
  title = {{AutoAssist: A Framework to Accelerate Training of Deep Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=4VMOY3b6Fd2},
}
Figure