Training deep neural networks (DNNs) is a resource-intensive process. While first-order optimizers like SGD or Adam are standard, researchers have long known that second-order methods—which utilize curvature information like the Hessian or Fisher Information Matrix—can converge in far fewer iterations. However, the massive matrix inversions required by these methods have historically been too computationally expensive for traditional CPUs and GPUs.
In their paper, "RePAST: A ReRAM-based PIM Accelerator for Second-order Training of DNN," Zhao et al. propose a specialized hardware architecture using Resistive RAM (ReRAM) and Process-In-Memory (PIM) techniques to make second-order training practical for large-scale models.
The Precision Bottleneck in Analog PIM
Standard first-order methods often follow a sub-optimal "zig-zag" path to the loss minimum. Second-order methods, such as K-FAC or Gauss-Newton, allow for larger, more accurate steps toward convergence. The primary challenge is the inversion of the Second-Order Information (SOI) matrix. For complex models like ResNet-50, the SOI matrix is enormous, and inversion has a computational complexity of $O(n^3)$.
While analog PIM accelerators excel at Vector-Matrix Multiplication (VMM), they struggle with high-precision matrix inversion. Analog circuits are inherently noisy, typically limiting them to 8-bit precision. As shown in the authors' evaluation, this level of precision is insufficient for stable DNN training.
As illustrated above, training loss explodes when using 8-bit or 12-bit precision for the SOI matrix. Modern DNN training requires at least 16-bit precision to maintain convergence and achieve high test accuracy.
The RePAST Solution: Algorithm and Architecture
To overcome the precision limits of analog hardware, RePAST employs a three-pronged approach:
- High-Precision Inversion Algorithm: The authors utilize a Taylor-expansion-based algorithm to decompose a high-precision (16-bit) inversion into a series of low-precision (8-bit) inversions and VMM operations. By iteratively refining the result, the system achieves the necessary precision using existing analog hardware components.
- Tiled Hardware Architecture: RePAST features a tiled layout where each "sub-tile" integrates VMM crossbars with a dedicated Matrix Inversion (INV) crossbar. This design allows for "fused" operations, computing complex expressions like $(A_1 \cdot A_2)^{-1}b$ directly within the analog domain, minimizing data movement to digital buffers.
- Software Mapping Scheme: A cost-function-based strategy analyzes the training algorithm's Dataflow Graph (DFG) to determine the most efficient hardware layout, balancing the benefits of operation fusion against potential latency trade-offs.
Key Results and Performance
The RePAST accelerator demonstrates significant improvements over both traditional GPUs and existing PIM solutions:
- Massive Speedups: Compared to an NVIDIA Tesla V100 GPU, RePAST achieves an average 115.8x speedup and 41.9x better energy efficiency for second-order training.
- Beating First-Order PIM: RePAST is 11.4x faster than PipeLayer (a state-of-the-art PIM accelerator for standard first-order training) because it leverages the faster convergence of second-order algorithms.
- Hardware Longevity: By reducing the number of iterations needed for convergence, RePAST decreases ReRAM "write" operations by 55.7%, significantly extending the hardware's lifespan.
Conclusion and Future Outlook
RePAST represents a major advancement in making second-order DNN training feasible. By combining a clever iterative algorithm with a specialized PIM architecture, it bypasses the traditional precision limits of analog computing. While challenges remain regarding extreme-scale models and physical chip manufacturing, RePAST provides a clear path toward more efficient and faster deep learning training hardware.
Summary updated