Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference

Yida Wang, Haichen Shen, Jared Roesch, Zhi Chen, Wei Chen, Yong Wu, Mu Li, Vin Sharma, Zachary Tatlock, Yida Wang 0003

TL;DR

Nimble is an end-to-end compiler and runtime system designed to optimize neural networks with dynamic features like variable tensor shapes and control flow. By introducing a dynamic type system and a specialized virtual machine, Nimble achieves up to 20x speedups over traditional frameworks across CPU and GPU platforms.

What Problem Does This Paper Address?

Most deep learning compilers (like XLA or TVM) are built on the "static graph" assumption: they expect the model architecture and tensor shapes to be fixed at compile-time. This allows for aggressive optimizations, such as pre-allocating all memory and fusing operators into highly efficient kernels.

However, modern AI—especially in Natural Language Processing (NLP)—is increasingly dynamic. Models like BERT handle variable sequence lengths, and architectures like Tree-LSTMs change their execution path based on the structure of the input data. When faced with this dynamism, traditional compilers often fail or fall back to "eager" execution modes (like standard PyTorch or TensorFlow), which are flexible but suffer from high overhead and sub-optimal performance. The gap Nimble fills is providing the optimization benefits of a compiler to the messy, unpredictable world of dynamic models.

Methods

Nimble introduces several novel components to bridge the gap between static optimization and dynamic execution:

  • Dynamic Type System: The authors introduce an Any dimension to the compiler's type system. This allows the compiler to track and propagate shape information even when specific dimensions are unknown until runtime. They use "sub-shaping" (similar to sub-typing in programming languages) to ensure that concrete shapes are valid instances of dynamic ones.
  • Shape Functions: For every operator, Nimble generates a "shape function" that runs at runtime to calculate the exact output dimensions based on the input.
  • Dynamic Memory Planning: Unlike static compilers that pre-allocate a fixed block of memory, Nimble makes memory allocation explicit in its Intermediate Representation (IR). It introduces instructions like alloc_storage and kill into the code itself, allowing the compiler to optimize and reuse memory buffers even when their sizes change.
  • Symbolic Codegen & Dispatch: To keep kernels fast, Nimble generates multiple versions of a kernel specialized for different "residues" (e.g., different remainders when a variable length is divided by a hardware tiling factor). At runtime, it dispatches the most efficient version for the current shape.
  • Virtual Machine (VM) Runtime: Instead of a simple graph executor, Nimble uses a lightweight, register-based VM. This VM handles the complex control flow (if/else, loops) and manages the execution of optimized kernels with minimal overhead.

Datasets

The authors evaluated Nimble using three popular models representing different types of dynamism:

  • LSTM: Representing dynamic control flow (looping over sequences).
  • Tree-LSTM: Representing dynamic data structures, evaluated using the Stanford Sentiment Treebank (SST).
  • BERT: Representing dynamic tensor shapes (variable sequence lengths), evaluated using the Microsoft Research Paraphrase Corpus (MRPC).

Testing was performed across three hardware tiers: Intel Skylake CPUs, Nvidia Tesla T4 GPUs, and ARM Cortex A72 CPUs (representing edge devices).

Key Insights and Ideas

  • Internalizing Memory Management: The "aha moment" here is moving memory management from a hidden runtime task into the compiler's IR. By making alloc and free visible to the compiler, Nimble can apply static-style optimizations (like buffer sharing) to dynamic workloads.
  • CPU-GPU Co-design for Shapes: Nimble intelligently places shape functions on the CPU even when the main kernels run on the GPU. This prevents the GPU from idling while waiting for complex shape logic to resolve.
  • The Power of a Specialized ISA: By creating a custom Instruction Set Architecture (ISA) for the VM that includes "super-instructions" (coarse-grained operations like InvokePacked), Nimble reduces the number of branches the CPU has to predict, significantly lowering runtime overhead compared to traditional interpreters.

Key Results

Nimble consistently outperformed baselines like PyTorch, MXNet, and DyNet:

  • Speedups: Nimble achieved latency reductions ranging from 1.05x to 19.9x compared to the best alternative on various platforms.
  • Tree-LSTM Performance: On Intel CPUs, Nimble was 17.4x faster than PyTorch; on ARM CPUs, it was 19.8x faster. This is largely because Nimble compiles the tree-traversal logic into efficient VM instructions rather than relying on slow Python-based orchestration.
  • Memory Efficiency: The dynamic memory planning pass reduced the number of buffer allocations by 47% and lowered memory allocation latency by 75% on Intel CPUs.
  • Portability: Unlike framework-specific optimizations, Nimble provides a unified approach that works across cloud CPUs, GPUs, and edge devices without needing to rewrite model logic.

Limitations and Future Work

  • Inference Focus: The current paper focuses strictly on model inference. The authors note that extending these dynamic optimizations to training is a key area for future research.
  • Accelerator Support: While it works on CPUs and GPUs, the authors aim to expand support to specialized AI accelerators (TPUs, NPUs), which often have even stricter requirements for static shapes.
  • Auto-tuning Complexity: Tuning kernels for symbolic shapes is more complex than for static ones. While Nimble uses a template-based approach, the authors suggest that more advanced machine learning-based auto-tuning could further close the performance gap between symbolic and static kernels.

Summary updated

Abstract

Modern deep neural networks increasingly make use of features such as dynamic control flow, data structures and dynamic tensor shapes. Existing deep learning systems focus on optimizing and executing static neural networks which assume a pre-determined model architecture and input data shapes--assumptions which are violated by dynamic neural networks. Therefore, executing dynamic models with deep learning systems is currently both inflexible and sub-optimal, if not impossible. Optimizing dynamic neural networks is more challenging than static neural networks; optimizations must consider all possible execution paths and tensor shapes. This paper proposes Nimble, a high-performance and flexible system to optimize, compile, and execute dynamic neural networks on multiple platforms. Nimble handles model dynamism by introducing a dynamic type system, a set of dynamism-oriented optimizations, and a light-weight virtual machine runtime. Our evaluation demonstrates that Nimble outperforms state-of-the-art deep learning frameworks and runtime systems for dynamic neural networks by up to 20x on hardware platforms including Intel CPUs, ARM CPUs, and Nvidia GPUs.
BibTeX
@inproceedings{wang2021nimble,
  author = {Yida Wang and Haichen Shen and Jared Roesch and Zhi Chen and Wei Chen and Yong Wu and Mu Li and Vin Sharma and Zachary Tatlock and Yida Wang},
  title = {{Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference}},
  booktitle = {Mlsys},
  year = {2021},
}

Loading related content...

Figure