CAFIN: CUDA-accelerated Adaptive Fisher Information Matrix Network Optimization via GPU-Native Kernel Fusion

Maryam Arshad, Muhammad Khuram ShahzadarXiv 2026 · arXiv

TL;DR

CAFIN introduces a GPU-native kernel fusion framework designed to accelerate the AdaFisher second-order optimizer. By collapsing multiple sequential CUDA kernels into a single, integrated operation, the authors achieve up to a 78.4% reduction in Fisher Information Matrix (FIM) construction latency and significant memory savings without altering the optimizer's mathematical behavior or convergence accuracy.

What Problem Does This Paper Address?

Deep learning models are typically trained using first-order optimizers like SGD or Adam, which rely solely on gradient information. Second-order optimizers, which incorporate "curvature" information (via the Hessian or Fisher Information Matrix), can converge faster and more reliably in complex loss landscapes. However, they are rarely used in practice because they are computationally expensive and memory-intensive.

Even efficient approximations like AdaFisher suffer from a "framework tax." In standard PyTorch implementations, the optimizer’s steps are executed as a sequence of separate operations (e.g., Kronecker products, additions, moving average updates). Each operation launches a new CUDA kernel and writes intermediate results back to the GPU's global memory (DRAM). This creates massive overhead due to kernel dispatch latency and redundant memory traffic—a classic "memory wall" problem where the GPU spends more time moving data than actually calculating.

Methods

The authors propose CAFIN (CUDA-accelerated Adaptive Fisher Information Matrix), which uses progressive kernel fusion to streamline the optimizer. Instead of six separate kernel launches per layer, CAFIN integrates these steps into a single "GPU-native" kernel.

The fusion happens in three stages:

  1. Fused Preconditioner Construction: Combines the outer-product and damping operations into one kernel, eliminating intermediate tensor materialization.
  2. Fused EMA + Preconditioner: Integrates Exponential Moving Average (EMA) updates with the preconditioner construction.
  3. Fully Fused Optimizer Step: The final stage, which merges EMA updates, FIM reconstruction, weight decay, momentum accumulation, and parameter updates into one single CUDA kernel per layer.
Fig. 2. Comparison of execution flow between baseline and fully fused AdaFisher optimizers. The baseline uses multiple CUDA kernels with intermediate global memory transfers, while the fully fused implementation combines all operations into a single kernel, leveraging registers and shared memory to minimize DRAM traffic and kernel-launch overhead.

Key technical optimizations include:

  • Shared-Memory Reuse: Loading EMA vectors into on-chip shared memory so all threads in a block can access them without hitting DRAM.
  • Register-Resident Computation: Computing the FIM elements directly in registers and consuming them immediately for the parameter update, meaning the full FIM matrix is never actually written to VRAM.

Datasets

The implementation was evaluated using a ResNet-18 architecture across three standard image classification benchmarks:

  • CIFAR-10
  • CIFAR-100
  • Tiny ImageNet

Testing was performed on consumer-grade hardware, specifically the NVIDIA GTX 1660 Ti (Turing architecture) and RTX 3060 Laptop (Ampere architecture).

Key Insights and Ideas

  • The Memory-Bound Reality: Through roofline analysis, the authors show that FIM reconstruction is a low-arithmetic-intensity workload. The bottleneck isn't the number of calculations (FLOPs), but the memory bandwidth. Fusion increases arithmetic intensity, moving the operation closer to the hardware's theoretical performance ceiling.
  • Zero VRAM Allocation for FIM: One of the most elegant parts of CAFIN is that by computing FIM elements "on-the-fly" in registers, it eliminates the need to store the full $d_{out} \times d_{in}$ matrix in global memory.
  • Amdahl’s Law in Action: While the optimizer is only one part of the training pipeline (backward propagation remains the biggest bottleneck), accelerating it still yields meaningful real-world gains. A 78% speedup in FIM construction translates to saving over an hour in a typical 18-hour training run.
  • Numerical Equivalence: The authors prove that fusion doesn't mean "approximation." By measuring the relative Frobenius norm of parameter updates, they confirmed that CAFIN stays within a strict numerical tolerance ($2 \times 10^{-6}$) compared to the baseline PyTorch implementation.
Layer-wise numerical error showing that CAFIN remains well below the predefined tolerance threshold.

Key Results

  • Latency Reduction: Isolated FIM construction latency dropped by up to 78.4%. Total optimizer-step latency was reduced by 2.1% to 26.9%, depending on the hardware and layer configuration.
  • Memory Savings: The fused implementation eliminated approximately 18.91 MB of intermediate GPU memory allocation per optimizer step for ResNet-18.
  • Throughput: On the GTX 1660 Ti, the fused preconditioner achieved ~1.59 GFLOPS, a significant jump from the 0.31 GFLOPS of the naive baseline.
  • Convergence: Training curves for CIFAR-10 and CIFAR-100 showed that CAFIN maintains the same accuracy and convergence behavior as the original AdaFisher.

Limitations and Future Work

  • Precision Support: The current study is limited to FP32. The authors note that extending CAFIN to mixed-precision (FP16/BF16) is a priority for future work.
  • Scalability: The experiments were conducted in a single-GPU environment. The impact of fusion on distributed multi-GPU training (where synchronization overhead is a factor) remains to be seen.
  • Model Diversity: While ResNet-18 is a standard benchmark, the authors acknowledge that Transformer-based architectures have different memory access patterns and may require different tiling strategies for optimal fusion.
  • Scope: This work focuses purely on execution-level acceleration; it does not attempt to improve the underlying mathematical "quality" of the AdaFisher optimizer itself.

Summary updated

Abstract

Second-order optimizers improve deep neural network training using curvature information, but their practical efficiency is limited by GPU execution overhead. AdaFisher reduces Fisher Information Matrix (FIM) approximation costs through diagonal Kronecker structures and exponential moving averages, yet its PyTorch implementation requires sequential CUDA kernel launches, intermediate tensor allocations, and repeated memory accesses. We introduce CAFIN, a GPU-native CUDA kernel fusion framework that improves AdaFisher execution efficiency without changing its optimization formulation. The proposed approach progressively integrates FIM preconditioner construction, exponential moving average updates, gradient preconditioning, momentum accumulation, weight decay, and parameter updates into fused CUDA kernels. Shared-memory reuse, register-resident computation, and in-place operations further reduce memory traffic and intermediate allocations. Experiments using ResNet-18 on CIFAR-10, CIFAR-100, and Tiny ImageNet across NVIDIA GTX 1660 Ti and RTX 3060 Laptop GPUs demonstrate up to 78.4% lower isolated FIM construction latency and 2.1-26.9% lower optimizer-step latency. The implementation also eliminates approximately 18.91 MB of intermediate GPU memory allocations per optimizer step. Numerical validation confirms consistency within the predefined FP32 tolerance, while convergence accuracy remains comparable across evaluated datasets and random seeds. These findings demonstrate that CUDA kernel fusion can substantially accelerate second-order optimization while preserving AdaFisher's underlying mathematical formulation and learning behavior.
BibTeX
@misc{arshad2026cafin,
  author = {Maryam Arshad and Muhammad Khuram Shahzad},
  title = {{CAFIN: CUDA-accelerated Adaptive Fisher Information Matrix Network Optimization via GPU-Native Kernel Fusion}},
  year = {2026},
  eprint = {2611.14589},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14589},
}
Figure