Compiler Integrated Automatic Differentiation and Differentiable Programming

This direction is split into other directions: higher-order automatic differentiation. Its last version stays readable and citable here.

Version 1 of 1172 papers379 conceptsfrozen 2026-10-04

In the early days of deep learning, practitioners faced a rigid choice: the flexibility of "define-by-run" frameworks like PyTorch or the performance of "define-then-run" graph compilers like TensorFlow. Modern research has collapsed this dichotomy by treating Automatic Differentiation (AD) not as a library-level trick, but as a fundamental compiler transformation.

By integrating AD directly into the compiler toolchain, researchers are moving toward Differentiable Programming, where any program—containing loops, recursion, and complex data structures—can be transformed into an efficient gradient-calculating version of itself.

Beyond the Global Tape

Traditional AD libraries often rely on a "tape" or "Wengert list" to record operations during execution. While intuitive, this approach struggles with complex control flow. As Fei Wang and Tiark Rompf argue in A Language and Compiler View on Differentiable Programming, managing this external state introduces significant overhead and complicates optimizations for nested loops or conditional logic.

Instead of recording a history of operations, Source Code Transformation (SCT) systems like Tangent, developed by Bart van Merrienboer and Alexander B. Wiltschko, apply adjoint rules directly to a program's Abstract Syntax Tree (AST). This generates standalone, readable source code for the backward pass. This approach allows standard compiler optimizations—like dead code elimination and algebraic simplification—to be applied to the gradient logic itself, often resulting in performance comparable to hand-optimized kernels van Merrienboer et al..

Differentiating the Intermediate Representation

While SCT works at the high-level source code, a more powerful approach is to perform AD on the compiler's Intermediate Representation (IR). This is the core insight behind Enzyme, introduced by William S. Moses and Valentin Churavy.

By operating on LLVM IR, Enzyme becomes language-agnostic, capable of synthesizing gradients for code written in C, C++, Fortran, or Rust Moses and Churavy. More importantly, it allows the compiler to optimize the "primal" (forward) code before differentiation. For example, if a compiler simplifies a loop via loop-invariant code motion, Enzyme differentiates the simplified version. This ensures that the resulting gradient code is mathematically equivalent but computationally leaner than what could be produced from raw source code.

Similarly, the DLVM infrastructure proposed by Richard Wei treats AD as a first-class compiler pass within a modular, SSA-based IR. This centralization allows the same optimization pipeline to handle both the forward and backward passes, simplifying the creation of new differentiable languages Wei et al..

Functional Foundations and Closures

A major challenge in AD is "remembering" values from the forward pass for use in the backward pass. In functional programming contexts, this is solved by having each function return its result along with a backpropagator closure.

In the Myia toolchain, Bart van Merrienboer and Olivier Breuleux demonstrate that this closure-based approach avoids the need for a global tape. Each closure captures only the specific local environment and intermediate values it needs. This functional representation allows AD to naturally handle recursion and higher-order functions—constructs that typically baffle standard deep learning frameworks van Merrienboer et al..

This philosophy is extended to the Julia ecosystem via Zygote.jl. As Mike Innes and Alan Edelman explain, integrating AD as a first-class language feature allows researchers to differentiate through existing scientific libraries and complex simulations without rewriting them in a domain-specific language Innes et al..

Efficiency on the Metal

Compiler-integrated AD isn't just about abstraction; it's about raw efficiency. By leveraging Just-In-Time (JIT) compilation, tools like ForwardDiff in Julia can generate specialized machine code for derivatives at runtime. Jarrett Revels and Miles Lubin showed that by stack-allocating "dual numbers" in fixed-size chunks, forward-mode AD can achieve performance competitive with low-level C++ Revels and Lubin.

This efficiency is critical for resource-constrained environments. For instance, the Tiny Training Engine (TTE) developed by Ji Lin and colleagues at MIT uses compile-time AD and graph pruning to enable on-device training on microcontrollers with less than 256KB of memory. By shifting the gradient calculation to the compilation phase, they eliminate the memory overhead of heavy runtime libraries, making "bare-metal" lifelong learning possible Lin et al..

The Future: Automatic Symbolic Optimization

The next frontier involves merging AD with symbolic rewriting. Felix Dangel has shown that by representing operations like convolutions as Tensor Networks or using Collapsed Taylor Mode, compilers can apply automated graph rewrites to simplify high-order derivatives like Laplacians. These optimizations, which exploit mathematical linearity during the propagation of derivatives, can provide massive speedups over standard nested backpropagation Dangel et al..

By treating differentiation as a first-class citizen of the compiler, the field is moving toward a world where the boundary between "writing code" and "optimizing a model" entirely disappears.

Papers in this version

Showing 60 of 172 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compiler_integrated_automatic_differenti_7943ec13,
  title = {Compiler Integrated Automatic Differentiation and Differentiable Programming},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compiler-integrated-automatic-differentiation-and-differentiable-programming/rdr_7943ec133d724506a57d40da9b79fcf5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compiler_integrated_automatic_differenti_7943ec13_v1,
  title = {Compiler Integrated Automatic Differentiation and Differentiable Programming},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compiler-integrated-automatic-differentiation-and-differentiable-programming/rdr_7943ec133d724506a57d40da9b79fcf5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}