Abstract
Deep Reinforcement Learning (DRL) is traditionally computationally intensive, requiring high-precision floating-point arithmetic. FIXAR is a novel hardware/software co-design platform that enables efficient DRL training and inference using fixed-point arithmetic on FPGAs. By employing quantization-aware training (QAT) and adaptive parallelism, FIXAR achieves a 2.7x throughput increase and 15.4x better energy efficiency compared to high-end CPU-GPU systems, without compromising algorithmic accuracy.
Introduction
The training of DRL agents is a resource-heavy process, typically necessitating high-end GPUs and power-hungry 32-bit floating-point (FP32) calculations. While supervised learning has successfully adopted quantization—using lower-precision integers to reduce power consumption—DRL has remained reliant on FP32 due to its extreme sensitivity to noise. In DRL, early-stage quantization noise can lead to poor decision-making, preventing the agent from exploring optimal paths and causing training failure.
To address these challenges, researchers from KAIST developed FIXAR, the first platform to bring fixed-point efficiency to the entire DRL pipeline.
Methodology: HW/SW Co-Design
FIXAR utilizes a two-pronged approach to maintain accuracy while maximizing efficiency:
1. Quantization-Aware Training (QAT)
Instead of immediate low-precision execution, FIXAR starts training with 32-bit fixed-point precision. After a "quantization delay" allows the model to stabilize, the system dynamically reduces activation precision to 16-bit. This "warm-up" period ensures the agent establishes a stable policy before introducing the noise associated with lower bit-widths.
2. Adaptive FPGA Accelerator
The hardware architecture is centered on Adaptive Array Processing (AAP) cores designed for flexibility.
- Configurable Datapath: Processing Elements (PEs) can switch between performing one 32-bit multiplication or two 16-bit multiplications simultaneously, effectively doubling throughput during the 16-bit phase.
- Adaptive Parallelism: The hardware dynamically adjusts its parallelization strategy based on the workload. For inference (single vector), it employs "intra-layer parallelism." For training (batches), it switches to "intra-batch parallelism" to process multiple samples concurrently.
- On-Chip Memory Optimization: By fitting all weights and gradients into 1.05MB of on-chip BRAM, FIXAR eliminates the energy-intensive data transfers to external DRAM.
Experimental Evaluation
The platform was evaluated using the MuJoCo physics engine, focusing on continuous control tasks such as HalfCheetah, Hopper, and Swimmer. The Deep Deterministic Policy Gradient (DDPG) algorithm served as the primary benchmark.
Key Results
- Throughput: The FIXAR platform achieved 25,293.3 Inferences Per Second (IPS), outperforming a Titan RTX GPU setup by 2.7x.
- Efficiency: The FPGA accelerator demonstrated 15.4x higher energy efficiency than the GPU, with a power draw of ~20.4W compared to the GPU's ~56.7W.
- Algorithmic Integrity: Agents trained on FIXAR reached reward levels identical to those achieved with 32-bit floating-point implementations, proving that fixed-point math is viable for DRL.
Discussion and Future Work
FIXAR demonstrates that the "warm-up" period is critical for DRL to survive low-precision noise. By proving the viability of fixed-point math, this research paves the way for smaller, more efficient RL chips suitable for edge devices and robotics.
However, current model size constraints (limited by on-chip BRAM) mean that scaling to vision-based DRL will require advanced external memory management. Future iterations may also explore automated triggers for quantization delay and broader algorithmic compatibility with PPO or SAC.
Summary updated