Hardware-Efficient Transformer Training via Piecewise Affine Operations

Atli Kosson, Martin Jaggi

Abstract

Modern Transformer models require trillions of floating-point multiplications, creating a significant "energy tax" for deep learning. This paper presents a method for training Transformers without standard floating-point multiplications by utilizing Piecewise Affine Operations (PAO). By treating the bit-level integer representations of floating-point numbers as operands for addition, we approximate multiplication and other elementary functions. Our approach covers the entire training loop, including the forward pass, backward pass, and optimizer. Experimental results on ImageNet-1k and IWSLT14 translation tasks show that our multiplication-free models achieve near-baseline accuracy while offering a theoretical reduction in hardware energy costs of up to 95%.

Introduction

The computational cost of training state-of-the-art deep learning models is dominated by floating-point multiplications. In standard hardware, a float32 multiplication consumes approximately four times more energy than a corresponding addition. While previous research, such as AdderNets, attempted to replace multiplications with additions, these methods often required specialized architectural modifications or complex backpropagation heuristics to maintain stability.

This work introduces a "drop-in" replacement for multiplication that is compatible with the standard Transformer pipeline. By leveraging the structure of IEEE 754 floating-point representations, we define a suite of Piecewise Affine Operations that allow for fully multiplication-free training without sacrificing model performance.

Methodology

Piecewise Affine Multiplication (PAM)

The core of our approach is Piecewise Affine Multiplication (PAM). Floating-point numbers are typically stored in memory using the IEEE 754 format: $(-1)^S \cdot 2^E \cdot (1+M)$, where $S$ is the sign bit, $E$ is the exponent, and $M$ is the mantissa. By treating these bit representations as integers and adding them, we effectively sum their exponents and approximate the addition of their logarithms. This operation is piecewise affine and can be implemented using simple integer addition and bit-shifting.

Beyond Matrix Multiplication

We extend this logic to define piecewise affine versions of other essential operations, including division ($1/x$), square root ($\sqrt{x}$), exponentiation ($\exp$), and logarithms ($\log$). This allows for the replacement of non-linearities like Softmax and LayerNorm with piecewise affine counterparts.

Piecewise affine approximations of common mathematical functions.

Mimic Derivatives

A significant hurdle in training with piecewise functions is that their exact derivatives are often discontinuous or zero, leading to unstable gradients. To address this, we employ "Mimic Derivatives." Instead of using the exact derivative of the piecewise function, we approximate the analytical derivative of the original operation (e.g., standard multiplication) using piecewise affine logic. This ensures stable convergence during backpropagation.

Multiplication-Free Optimizer

To achieve a truly multiplication-free training loop, we also replace the multiplications and divisions within the AdamW optimizer. This results in a system where the entire training process—from data processing to weight updates—is free of standard floating-point multipliers.

Experimental Setup

We evaluated the proposed method across several standard benchmarks:

  • Vision: We trained the DeiT-Tiny (Vision Transformer) architecture on ImageNet-1k and CIFAR-10.
  • Language: We performed German-to-English translation using the IWSLT14 DE-EN dataset with a Transformer-Small model.
  • Robustness: Additional tests were conducted on convolutional architectures, including VGG-13, ResNet, and ConvMixer on CIFAR-10.

Key Results

Accuracy and Performance

Our results indicate that the "multiplication tax" is not a requirement for high-performance deep learning. On the ImageNet-1k benchmark, the multiplication-free DeiT-Tiny model achieved an accuracy of 72.2%, perfectly matching the standard floating-point baseline. On CIFAR-10, the PAM-based model slightly outperformed the baseline (95.0% vs 94.8%).

Performance comparison of PA-Matmul against baseline and AdderNet on vision benchmarks.

In language tasks, the model reached a BLEU score of 34.2 on the IWSLT14 dataset, representing a negligible drop from the 34.4 baseline.

Hardware Efficiency

The theoretical benefits of this approach are substantial. We estimate that a dedicated hardware unit for Piecewise Affine Operations would require only 5.4% of the energy and 3.6% of the silicon area compared to a standard float32 multiplier. Furthermore, we found that the mantissa can be truncated to as few as 4 bits without significant loss in accuracy, suggesting further potential for memory bandwidth optimization.

Discussion and Limitations

While the theoretical energy savings are immense, a "software gap" currently exists. Modern GPUs are highly optimized for standard floating-point arithmetic. Without native hardware support for PAM instructions, our custom CUDA kernels are currently slower than standard matrix multiplications. Real-world speedups will require implementation on custom ASICs or FPGAs.

Additionally, while our methods scaled well to "Tiny" and "Small" models, further research is needed to determine if these approximation errors compound in massive models with hundreds of billions of parameters. The sensitivity of the loss function to early backpropagation errors also remains an area for future investigation.

Conclusion

This work demonstrates that modern Transformers can be trained entirely without standard multiplications. By utilizing Piecewise Affine Operations and Mimic Derivatives, we achieve baseline-level accuracy while paving the way for a new generation of ultra-energy-efficient AI hardware.

Summary updated

BibTeX
@inproceedings{kosson2023hardware,
  author = {Atli Kosson and Martin Jaggi},
  title = {{Hardware-Efficient Transformer Training via Piecewise Affine Operations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=Vtqymej1tA},
}
Figure