This paper from the Google Brain team presents a significant leap in automating the distribution of deep learning models across hardware. By introducing a hierarchical reinforcement learning (RL) model, the authors automate the partitioning and placement of massive computational graphs (up to 80,000+ operations) across heterogeneous hardware devices. This approach achieves up to a 60.6% runtime speedup over human-expert placements for complex models like Neural Machine Translation (NMT).
The Challenge of Device Placement
As deep learning models grow in complexity, they frequently exceed the memory and processing power of a single GPU. To train these models efficiently, developers must distribute the "computational graph"—the network of mathematical operations—across multiple devices.
Historically, this distribution has relied on two suboptimal methods:
- Human Intuition: Experts manually assign layers to specific GPUs. This is time-consuming and often misses non-obvious, highly efficient optimizations.
- Algorithmic Partitioning: Traditional graph-cutting tools (like MinCut) attempt to balance computational load while minimizing communication overhead. However, these tools struggle with the dynamic nature of deep learning, where execution time depends heavily on the specific hardware and software stack.
Previous RL-based attempts worked well for small graphs but failed to scale beyond 1,000 nodes. The search space for deciding a device for every single operation in a modern network becomes mathematically astronomical.
A Hierarchical Approach: Grouper and Placer
To solve the scaling issue, the authors introduce a two-level hierarchical architecture trained end-to-end using policy gradients (REINFORCE).
The model consists of two primary components:
- The Grouper: A feed-forward network that takes "operation embeddings" (data about the operation type, its neighbors, and its output shapes) and assigns each operation to one of $M$ groups. This effectively compresses a massive graph of 80,000 nodes into a manageable number of clusters (e.g., 256).
- The Placer: A sequence-to-sequence (seq2seq) model equipped with an attention mechanism. It takes the embeddings of the groups created by the Grouper and predicts which physical device (e.g., CPU, GPU 0, GPU 1) should handle each group.
The Reward Loop: Unlike many RL tasks that rely on simulated environments, this model is trained on physical hardware. The predicted placement is executed, and the "reward" sent back to the RL agent is the negative square root of the actual execution time. Faster placements reward the agent, while slower ones or crashes (like Out of Memory errors) penalize it.
Key Results and Insights
The authors evaluated their model on four heavy-hitting TensorFlow architectures: Inception-V3, ResNet-152, RNNLM, and NMT (up to 8 layers and 83,712 operations).
By jointly training the Grouper and Placer, the model learns to cluster operations not just by proximity, but in a way that optimizes the Placer's subsequent device assignment. This reduces the search space from $D^N$ (where $N$ is operations) to $D^M$ (where $M$ is groups), making RL viable for production-scale models.
The hierarchical model consistently outperformed human experts and traditional algorithms:
- Neural Machine Translation (NMT): Achieved a 60.6% reduction in runtime per training step for a 2-layer model and a 53.7% reduction for a 4-layer model compared to human experts. The RL agent discovered non-trivial placements, such as interleaving different steps of the same layer across multiple GPUs—a strategy too complex for manual human design.
- Inception-V3: Found a placement 16.3% faster than the best human-designed strategy.
- ResNet & RNNLM: The model correctly learned that for these specific architectures, the most efficient placement was a single GPU to minimize communication overhead, matching the human expert baseline.
While training the RL agent takes a few hours, the GPU-hours saved by the optimized placement during weeks of model training far outweigh the initial overhead.
Limitations and Future Work
Currently, the model assumes the computational graph remains static during training. Furthermore, training the RL agent requires running the target model many times on real hardware to measure speed, necessitating a dedicated cluster of GPUs. Future work may explore training the placement RL agent simultaneously with the target model itself, allowing the placement strategy to adapt dynamically as the model's weights evolve.
Summary updated