TL;DR
Nimble is a lightweight execution engine that dramatically speeds up deep learning by eliminating CPU-side scheduling overhead and automatically parallelizing tasks on the GPU. By moving scheduling logic "ahead-of-time" and intelligently distributing tasks across multiple GPU streams, Nimble achieves up to a 22x speedup in inference and a 3.6x speedup in training compared to PyTorch.
What Problem Does This Paper Address?
Modern GPUs are incredibly fast, but they often sit idle while waiting for the CPU to tell them what to do next. This "scheduling overhead" includes checking tensor shapes, allocating memory, and dispatching kernels. In many state-of-the-art neural networks—especially those designed by Neural Architecture Search (NAS)—the individual computation tasks are small and numerous. In these cases, the CPU can take longer to schedule a task than the GPU takes to execute it.
The paper highlights a startling reality: in some models, the GPU is active only 9% of the time. The rest of the time is spent waiting on the framework's runtime stack. Furthermore, most deep learning frameworks (like PyTorch and TensorFlow) execute tasks serially—one after another. However, many modern networks have "branches" that could theoretically run at the same time. Because current frameworks don't effectively use multiple GPU streams, they miss out on significant potential for parallel execution.
Methods
The authors introduce two primary innovations to solve these bottlenecks, treating scheduling much like "loop-invariant code motion" in a compiler to move work out of the critical execution loop.
- Ahead-of-Time (AoT) Scheduling: Nimble observes that for most production models, the network structure and input shapes are static. Instead of re-calculating shapes and memory requirements every single time a model runs, Nimble performs a "pre-run." During this phase, it records a trace of every GPU kernel and memory request. At runtime, it simply "replays" this trace, bypassing the heavy CPU logic entirely.
- Automatic Multi-stream Execution: To exploit parallelism, Nimble includes a Graph Rewriter. It analyzes the model's computation graph and assigns operators to different GPU streams (independent execution queues).
- Optimal Stream Assignment Algorithm: Assigning tasks to streams manually is a nightmare for developers. Nimble uses a novel algorithm based on graph theory (specifically, finding a minimum equivalent graph and a maximum bipartite matching) to find a schedule that maximizes parallel execution while minimizing the number of expensive "synchronization" events between streams.
Datasets and Evaluation
The researchers evaluated Nimble across a wide array of popular and complex neural network architectures:
- Standard Models: ResNet-50, ResNet-101, Inception-v3, MobileNetV2, and BERT.
- NAS-generated Models: NASNet-A (Mobile and Large), AmoebaNet, and DARTS.
- Optimized Models: EfficientNet (B0 and B5).
These models were tested using the ImageNet and CIFAR-10 datasets for inference and training benchmarks.
Key Results
- Massive Inference Speedups: Compared to PyTorch, Nimble is up to 22.34× faster for models with high parallelism like NASNet-A.
- Outperforming Specialized Engines: Nimble isn't just faster than general frameworks; it also beats specialized inference compilers like NVIDIA TensorRT (up to 2.81×) and Apache TVM (up to 1.70×).
- Training Gains: While training usually involves larger batches (which hides overhead), Nimble still provides a 3.61× speedup for training on smaller-scale inputs like CIFAR-10.
- Multi-stream Efficiency: The automatic multi-stream execution alone provided up to a 1.88× boost over Nimble's own single-stream performance, proving that inter-operator parallelism is a major untapped resource.
Limitations and Future Work
- Static Graph Constraint: Nimble is designed for static neural networks. It may not be directly applicable to models with highly dynamic control flow (like certain recursive neural networks) where the graph changes based on input values.
- Memory Rigidity: Because Nimble pre-allocates memory based on the AoT trace, it might be less flexible in environments where memory needs to be shared dynamically between different concurrent applications.
- Framework Integration: While currently built for PyTorch (allowing users to wrap existing models with minimal code changes), the authors note that the design is general enough to be applied to other frameworks like TensorFlow or MXNet in the future.
Summary updated