Spatially Parallel Convolutions

Peter H. Jin, Boris Ginsburg, Kurt KeutzerWorkshops @ ICLR 2018

Overview

This paper introduces spatially parallel convolutions, a technique designed to split large input tensors (such as high-resolution images) across multiple GPUs along their spatial dimensions. By adapting "halo region" and "ghost zone" concepts from traditional high-performance computing (HPC), the authors successfully bypass the memory limits of a single GPU. This approach enables the training of high-resolution models with near-linear—and occasionally superlinear—speedups.

The Memory Wall in High-Resolution Deep Learning

Deep learning models for tasks like semantic segmentation or high-resolution medical imaging require massive amounts of GPU memory. While high-end GPUs typically offer 12–16 GB of RAM, a single high-resolution image and its associated feature maps can easily exceed this limit.

Standard solutions to this memory bottleneck have notable drawbacks:

  • Data Parallelism: Splits the batch across GPUs, but provides no benefit if a single image is too large for one GPU.
  • Model Parallelism: Splits the network layers across GPUs, which can lead to complex synchronization and pipeline bubbles.
  • Checkpointing: Saves memory by recomputing activations during the backward pass, but significantly increases overall training time.

The authors aim to provide a way to scale "horizontally" across the image itself, allowing models to handle resolutions that were previously impossible to fit on a single device.

Methodology: Borrowing from HPC

The authors adapt a technique from stencil codes, which are commonly used in physics simulations and computational fluid dynamics.

  1. Spatial Partitioning: An input tensor is sliced into horizontal "stripes." Each stripe is assigned to a different GPU.
  2. Halo Regions and Ghost Zones: To perform a convolution at the edge of a slice, a GPU needs a few pixels from its neighbor (the "halo region"). Each GPU maintains a "ghost zone"—a small buffer of padded memory—to receive these pixels from adjacent GPUs.
  3. Forward Pass: GPUs exchange halo regions to fill their ghost zones, then perform standard convolutions independently.
  4. Backward Pass: This acts as the "transpose" of the forward pass. Gradients are computed locally, followed by a "halo reduction" (summing gradients at the boundaries) to ensure mathematical correctness for backpropagation.

Crucially, this approach is adjoinable, meaning it integrates seamlessly into standard gradient-based optimization like Stochastic Gradient Descent (SGD).

Key Insights

  • Cross-Pollination from HPC: The core insight is realizing that a convolution is essentially a stencil operation. Techniques used for decades in weather forecasting and fluid simulations are perfectly suited for scaling Convolutional Neural Networks (CNNs).
  • Memory Scaling: Memory usage per GPU scales almost perfectly inversely with the number of GPUs ($1/P$). This effectively turns a cluster of GPUs into one giant "virtual GPU" with massive memory capacity.
  • Superlinear Speedups: In some tests, 4 GPUs were more than 4x faster than a single GPU. This typically occurs because smaller data chunks fit better into the GPU's high-speed cache, or because the underlying library (cuDNN) finds more efficient kernels for smaller sub-problems.

Key Results

The authors evaluated their method on a Tesla V100 system with NVLink, using synthetic microbenchmarks to measure wall-clock time and memory usage across different spatial dimensions ($128 \times 128$ to $512 \times 512$). They also verified mathematical correctness by training a spatially parallel ResNet-18 on ImageNet, matching standard top-1 validation accuracy (69.5%).

Standard Convolutions

For standard convolutions, the speedup was nearly linear and highly memory-efficient.

Standard Convolution Performance

As shown in the table above, a $512 \times 512$ convolution that took 45.15 ms on 1 GPU took only 10.65 ms on 4 GPUs—a 4.2x superlinear speedup. Furthermore, memory usage dropped proportionally from 4096 MB on a single GPU to just 1044 MB per GPU when distributed across four devices.

Dilated Convolutions

Dilated convolutions are inherently harder to parallelize due to their larger halo regions, which require more communication between GPUs.

Dilated Convolution Performance

Despite the increased communication overhead, dilated convolutions still demonstrated strong scaling on larger problem sizes, achieving up to a 3.5x speedup on 4 GPUs for $512 \times 512$ inputs.

Limitations and Future Work

  • Simple Partitioning: The current implementation only partitions tensors into horizontal stripes (1D partitioning). Future work could explore 2D "grid" partitioning for even better scaling across larger clusters.
  • Communication Overhead: The implementation uses a distinct communication phase before computation. The authors suggest that "fusing" communication and computation (overlapping them) using low-latency interconnects like NVLink would further improve performance.
  • Framework Integration: As a preliminary implementation, integrating this directly into mainstream frameworks (like PyTorch or TensorFlow) as a standard layer type is a necessary next step for widespread adoption.

Summary updated

BibTeX
@inproceedings{jin2018spatially,
  author = {Peter H. Jin and Boris Ginsburg and Kurt Keutzer},
  title = {{Spatially Parallel Convolutions}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
}
Figure