Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference

Haichen Shen, Jared Roesch, Zhi Chen, Wei Chen, Yong Wu, Mu Li, Vin Sharma, Zachary Tatlock, Yida Wang

Abstract

Modern deep neural networks increasingly make use of features such as dynamic control flow, data structures and dynamic tensor shapes. Existing deep learning systems focus on optimizing and executing static neural networks which assume a pre-determined model architecture and input data shapes--assumptions which are violated by dynamic neural networks. Therefore, executing dynamic models with deep learning systems is currently both inflexible and sub-optimal, if not impossible. Optimizing dynamic neural networks is more challenging than static neural networks; optimizations must consider all possible execution paths and tensor shapes. This paper proposes Nimble, a high-performance and flexible system to optimize, compile, and execute dynamic neural networks on multiple platforms. Nimble handles model dynamism by introducing a dynamic type system, a set of dynamism-oriented optimizations, and a light-weight virtual machine runtime. Our evaluation demonstrates that Nimble outperforms state-of-the-art deep learning frameworks and runtime systems for dynamic neural networks by up to 20x on hardware platforms including Intel CPUs, ARM CPUs, and Nvidia GPUs.

Introduction

Deep learning compilers like XLA and TVM have traditionally relied on a "static graph" assumption, where model architectures and tensor shapes are fixed at compile-time. This allows for aggressive optimizations such as static memory allocation and operator fusion. However, modern NLP models like BERT and Tree-LSTMs introduce significant dynamism through variable sequence lengths and input-dependent execution paths. Traditional compilers often struggle with these features, falling back to "eager" execution modes that incur high runtime overhead. Nimble addresses this gap by providing a comprehensive system for optimizing and compiling dynamic neural networks.

System Architecture

Nimble consists of an end-to-end compiler and a specialized runtime environment. The compiler transforms high-level dynamic models into a platform-independent bytecode and platform-dependent kernels, which are then executed by a lightweight virtual machine.

The Nimble system architecture, illustrating the compiler pipeline and the runtime virtual machine.

The compiler pipeline includes several key stages:

  • Dynamic Type System: Introduces an Any dimension to track and propagate shape information even when specific dimensions are unknown until runtime.
  • Shape Functions: Generates functions for every operator to calculate output dimensions at runtime.
  • Dynamic Memory Planning: Explicitly manages memory within the Intermediate Representation (IR) using alloc_storage and kill instructions, enabling buffer reuse for dynamic sizes.
  • Symbolic Codegen: Produces specialized kernel versions for different shape "residues" to maintain high performance.

Key Optimizations and Device Placement

A critical challenge in dynamic networks is managing the overhead of shape calculations and data movement. Nimble employs intelligent device placement to optimize these tasks.

Device placement strategies: (a) shape functions on CPU, (b) explicit device copies, and (c) mutable invocation for dynamic workloads.

As shown in the figure above, Nimble often places shape functions on the CPU even when kernels execute on the GPU. This prevents GPU idling during complex shape resolution. Furthermore, by internalizing memory management into the IR, the compiler can apply static-style optimizations like buffer sharing to dynamic workloads, significantly reducing allocation overhead.

Evaluation

Nimble was evaluated using three representative dynamic models:

  • LSTM: Dynamic control flow via sequence looping.
  • Tree-LSTM: Dynamic data structures (evaluated on SST).
  • BERT: Dynamic tensor shapes (evaluated on MRPC).

Testing spanned Intel Skylake CPUs, Nvidia Tesla T4 GPUs, and ARM Cortex A72 CPUs.

Results

Nimble consistently outperformed established frameworks like PyTorch, MXNet, and DyNet:

  • Performance Gains: Latency reductions ranged from 1.05x to 19.9x. Notably, Nimble was 17.4x faster than PyTorch for Tree-LSTM on Intel CPUs.
  • Memory Efficiency: Dynamic memory planning reduced buffer allocations by 47% and lowered allocation latency by 75% on Intel CPUs.
  • Versatility: The system provides a unified approach across cloud and edge hardware without requiring model logic rewrites.

Conclusion

Nimble demonstrates that the benefits of high-performance compilation can be extended to dynamic neural networks. By combining a dynamic type system with a specialized VM and IR-level memory management, it achieves significant speedups over existing frameworks. Future work includes extending these optimizations to the training phase and expanding support for specialized AI accelerators.

Summary updated

Abstract

Modern deep neural networks increasingly make use of features such as dynamic control flow, data structures and dynamic tensor shapes. Existing deep learning systems focus on optimizing and executing static neural networks which assume a pre-determined model architecture and input data shapes--assumptions which are violated by dynamic neural networks. Therefore, executing dynamic models with deep learning systems is currently both inflexible and sub-optimal, if not impossible. Optimizing dynamic neural networks is more challenging than static neural networks; optimizations must consider all possible execution paths and tensor shapes. This paper proposes Nimble, a high-performance and flexible system to optimize, compile, and execute dynamic neural networks on multiple platforms. Nimble handles model dynamism by introducing a dynamic type system, a set of dynamism-oriented optimizations, and a light-weight virtual machine runtime. Our evaluation demonstrates that Nimble outperforms state-of-the-art deep learning frameworks and runtime systems for dynamic neural networks by up to 20x on hardware platforms including Intel CPUs, ARM CPUs, and Nvidia GPUs.
BibTeX
@inproceedings{shen2021nimble,
  author = {Haichen Shen and Jared Roesch and Zhi Chen and Wei Chen and Yong Wu and Mu Li and Vin Sharma and Zachary Tatlock and Yida Wang},
  title = {{Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference}},
  booktitle = {Mlsys},
  year = {2021},
}
Figure