DeepFlow: A Cross-Stack Pathfinding Framework for Distributed AI Systems

Newsha Ardalani, Saptadeep Pal, Puneet GuptaACM Trans. Design Autom. Electr. Syst 2022 · DOI · OpenReview

TL;DR

This paper introduces DeepFlow, a novel framework that enables cross-stack analysis from hardware technology to algorithmic design for distributed AI systems. It addresses the critical issue of low hardware utilization in large-scale AI training by providing a unified approach to explore the vast design space and automate co-optimization, leading to more efficient AI systems.

What Problem Does This Paper Address?

The paper tackles the growing problem of low hardware utilization (often 5-20%) in large-scale AI training systems. This inefficiency stems from a disconnect between different layers of the computing stack – from hardware micro-architecture and technology nodes to system architecture, network topology, and algorithmic design (model architecture, parallelism strategies). These layers are often developed independently by different teams or organizations, making it challenging to optimize the system holistically. As AI models become increasingly complex and demand more computational resources, this inefficiency becomes a major concern, hindering performance and driving up costs. The authors aim to bridge this gap by providing a framework that allows for unified analysis and optimization across the entire stack.

Methods

DeepFlow is built upon two core components:

  • CrossFlow: A performance modeling framework that enables "what-if" analysis across different layers of the stack. It uses an analytical model that considers the entire system architecture, offering a more sophisticated approach than simple Roofline analysis but being less time-consuming than full simulations. CrossFlow takes inputs like system topology, accelerator architecture templates, technology parameters, hardware design budgets, ML model compute graphs, and parallelism strategies to predict the runtime of an iteration.
  • DeepFlow (built on CrossFlow): This component uses machine learning-based techniques to automate the design space exploration and co-optimization process. It leverages a Search and Optimization Engine (SOE) that, inspired by gradient descent algorithms, navigates the complex design space to find optimal hardware-software-technology configurations.

The framework breaks down the problem into several modules:

  • Micro-Architecture Generator Engine (AGE): Generates micro-architectural parameters based on technology libraries, architecture templates, and hardware constraints (area, power, perimeter).
  • Compute Graph Transformation and Device Placement Engine (DPE): Transforms the ML model's compute graph to reflect parallelism strategies and maps these transformed graphs onto the system's hardware nodes, aiming to minimize communication overhead.
  • Performance Prediction Engine (PPE): Uses hierarchical roofline modeling and event-driven simulation to estimate the execution time of kernels and the overall end-to-end performance.

Datasets

The paper does not explicitly mention specific datasets used for training ML models. However, it validates its performance prediction model using:

  • Distributed GEMM (General Matrix Multiply) kernels: Across various shapes and parallelism strategies, run on NVIDIA P4 GPUs and NVIDIA DGX-1 systems.
  • Large-scale language models (LM): Specifically, a 2-layer LSTM model, evaluated on V100 GPUs.

The paper focuses on the performance characteristics of these workloads rather than the accuracy of the ML models themselves.

Key Insights and Ideas

  • Holistic Optimization is Crucial: The paper strongly emphasizes that optimizing individual layers of the stack in isolation is insufficient. True performance gains come from co-optimizing technology, hardware architecture, and parallelism strategies.
  • Technology Scaling Alone Isn't Enough: While advancements in logic technology nodes are important, they alone provide diminishing returns for large AI workloads. Significant performance improvements require advancements in memory bandwidth (like HBM) and inter-node networking.
  • Parallelism Strategy Matters Immensely: The choice of parallelism strategy can yield substantial performance gains, often more than naive strategies on next-generation hardware. DeepFlow's ability to explore this space is a key contribution.
  • Cross-Stack Communication is Key: DeepFlow acts as a common language for researchers and engineers across different domains (hardware, software, algorithms) to understand each other's needs and constraints.
  • Design Space is Vast and Complex: The paper highlights the enormous search space of potential hardware-software-technology configurations, underscoring the need for automated exploration tools like DeepFlow.

Key Results

  • High Prediction Accuracy: CrossFlow's performance prediction model achieved near-perfect correlation (0.996) with measurements on real hardware for GEMM kernels and language models, with a mean relative error as low as 8.9%.
  • Impact of Technology Scaling: The study showed that logic scaling provides benefits up to a certain point (N3 node), after which memory bandwidth and network technology become the primary bottlenecks for large language models.
  • Parallelism Strategy Gains: Optimizing the parallelism strategy alone can offer up to a 2x performance improvement.
  • Co-optimization Benefits: Co-optimizing parallelism strategy and hardware architecture provides significant benefits, especially for mature technology nodes.
  • Multi-Node Package Insights: Increasing the number of nodes in a package improves performance, but gains become marginal beyond 4 nodes, suggesting that ultra-large packages might not be cost-effective for scaling LLM training.

Limitations and Future Work

  • Assumptions in Modeling: The paper acknowledges that the conclusions drawn are dependent on the assumptions made about the system architecture and dataflow (e.g., homogeneous replication of nodes within a package). Radical differences in these aspects could change the findings.
  • Focus on Specific Workloads: The current validation is primarily on GEMM kernels and language models. Extending the framework to other AI applications is a stated future direction.
  • Future Work: The authors plan to extend DeepFlow's modeling capabilities to other applications beyond language models and GEMM kernels. They also envision DeepFlow being used by ML practitioners, system designers, and technology experts for informed decision-making and future technology development guidance.

Summary updated

Abstract

Over the past decade, machine learning model complexity has grown at an extraordinary rate, as has the scale of the systems training such large models. However there is an alarmingly low hardware utilization (5-20%) in large scale AI systems. The low system utilization is a cumulative effect of minor losses across different layers of the stack, exacerbated by the disconnect between engineers designing different layers spanning across different industries. We propose CrossFlow, a novel framework that enables cross-layer analysis all the way from the technology layer to the algorithmic layer. We also propose DeepFlow (built on top of CrossFlow using machine learning techniques) to automate the design space exploration and co-optimization across different layers of the stack. We have validated CrossFlow accuracy with distributed training on real commercial hardware and showcase several DeepFlow case studies demonstrating pitfalls of not optimizing across the technology-hardware-software stack for what is likely, the most important workload driving large development investments in all aspects of computing stack.
BibTeX
@article{ardalani2022deepflow,
  author = {Newsha Ardalani and Saptadeep Pal and Puneet Gupta},
  title = {{DeepFlow: A Cross-Stack Pathfinding Framework for Distributed AI Systems}},
  journal = {ACM Trans. Design Autom. Electr. Syst.},
  year = {2022},
  doi = {10.1145/3635867},
}
Figure