Compiler Transformations for Source to Source Automatic Differentiation

152 concepts68 papers1 proposal

To make software adaptive, programs must be able to compute gradients. As Barak A. Pearlmutter argues in Automatic Differentiation and Ubiquitous Adaptive Computing, integrating gradient-based optimization directly into general-purpose programming allows systems to learn and evolve. However, calculating gradients through arbitrary, long-running, or recursive code structures presents a major technical challenge.

Historically, machine learning frameworks have relied on operator overloading. In this paradigm, standard mathematical operations are modified to record their execution onto a runtime data structure known as a "tape" or "trace." While highly flexible, this approach isolates gradient calculation from the broader compilation pipeline, introducing significant runtime overhead and preventing cross-cutting optimizations.

To overcome these limitations, a powerful paradigm has emerged: compiler-level and source-to-source automatic differentiation (AD). Instead of tracking operations at runtime, these systems treat differentiation as a first-class compiler pass, transforming the program's structure before execution to synthesize highly optimized derivative code.


Moving Differentiation into the Compiler Pipeline

The core insight behind compiler-level AD is that a derivative function can be generated by parsing and transforming a program's structure directly. This can happen either at the Abstract Syntax Tree (AST) level or within the compiler's Intermediate Representation (IR).

As demonstrated by Bart van Merrienboer et al. in Tangent: Automatic differentiation using source-code transformation for dynamically typed array programming, source-code transformation parses the original program's AST and applies differentiation rules line-by-line. This generates a standalone derivative function that can be inspected, debugged, and optimized using standard tools, completely eliminating the runtime overhead of maintaining an execution tape.

[Operator Overloading (Runtime)]
Program Execution ---> Records Operations to Tape ---> Processes Tape in Reverse

[Source-to-Source (Compile-time)]
Source Code / IR ---> AST/IR Differentiation Pass ---> Optimized Gradient Code

Operating at the IR level offers an even greater advantage: it allows differentiation to happen in tandem with standard compiler optimization passes. In Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients, William S. Moses et al. show that synthesizing gradients after optimizations like loop-invariant code motion results in leaner, faster derivative code. Because the compiler differentiates a simplified representation of the logic, the resulting gradient functions are computationally optimized while remaining mathematically exact.

Furthermore, Richard Wei et al. demonstrate in DLVM: A modern compiler infrastructure for deep learning systems that integrating AD as a native compiler pass ensures that the generated adjoint code is subject to the exact same optimization pipeline as the forward pass, enabling downstream optimizations like kernel fusion and algebraic simplification.


Eliminating the Tape via Continuations

One of the hardest problems in source-to-source reverse-mode AD is managing the control flow of the backward pass without a global tape. When a program contains complex loops, branches, or recursion, the backward pass must traverse these structures in reverse order.

To solve this cleanly, researchers have turned to functional programming theory—specifically Continuation-Passing Style (CPS) and delimited continuations.

In Backpropagation with Callbacks: Foundations for Efficient and Expressive Differentiable Programming, Fei Wang 0046 et al. show that backpropagation can be implemented by treating the forward and backward passes as a sequence of function calls and returns. By using delimited continuations, the remaining execution path of a program is captured as a function. This allows the backward pass to be integrated directly into the program's native control flow as a chain of callbacks.

As explained by the authors in A Language and Compiler View on Differentiable Programming, this functional approach transforms high-level logic containing complex control flow into flat, recursive low-level code. It preserves the structure of the original program without manual loop unrolling or operation tracing, allowing the compiler to emit highly efficient gradient computations free from runtime library overhead.


Graph Rewriting and Hybrid Symbolic Coarsening

Once a computation is represented within a compiler framework, advanced graph-rewriting techniques can be applied to optimize the mathematical structure of the derivatives themselves. Pure algorithmic differentiation tracks every single scalar operation, which can introduce significant memory and tracking overhead.

To mitigate this, Guoqiang Zhang et al. introduce a "coarsening" optimization in Coarsening Optimization for Differentiable Programming. This technique identifies specific code segments and elevates them into a symbolic representation for mathematical simplification before re-integrating them into the algorithmic differentiation pipeline. By differentiating these coarsened chunks symbolically, the system bypasses the need to track numerous fine-grained operations and wrap intermediate data in tensor containers.

[Standard AD Tracking]
x ---> [Mul] ---> t1 ---> [Add] ---> t2 ---> [Sin] ---> y  (Tracks every step)

[Coarsened Hybrid AD]
x ---> [ Differentiated Symbolically as sin(a*x + b) ] ---> y  (Bypasses intermediate steps)

Similarly, graph rewrites can drastically improve the scaling of high-order derivatives. In Collapsing Taylor Mode Automatic Differentiation, Felix Dangel et al. demonstrate that high-order derivative computations can be optimized by capturing the initial computational graph and applying automated rewrites that exploit the linearity of derivative coefficients. By summing these coefficients during propagation rather than after, the compiler replaces standard nested differentiation with optimized linear aggregations, yielding massive reductions in both execution time and memory footprint.


Language Integration and Hardware Benefits

Integrating automatic differentiation deeply into a language's compiler infrastructure yields practical benefits that extend from high-level software design down to bare-metal hardware execution.

  1. Native Type Composition: In Generalized Physics-Informed Learning through Language-Wide Differentiable Programming, Christopher Vincent Rackauckas et al. show how a compiler-level AD system can leverage a language's native type system and multiple dispatch. This allows the system to compute gradients through complex, pre-existing codebases and user-defined data structures without requiring developers to rewrite their simulations inside specialized machine learning frameworks.
  2. Extreme Memory Constraints: Shifting gradient synthesis to compile-time is critical for resource-constrained environments. As Ji Lin et al. demonstrate in On-Device Training Under 256KB Memory, compile-time AD produces a static execution graph that generates slim binary code. This removes the heavy memory footprint of runtime tracking libraries, making it possible to perform on-device training directly on microcontrollers with less than 256KB of SRAM.

By moving automatic differentiation out of runtime libraries and treating it as a fundamental compiler transformation, the field bridges the gap between mathematical expressiveness and execution performance, paving the way for highly optimized, adaptive computing systems.

Key Foundational Work

An Introduction to Using Software Tools for Automatic Differentiation (2003) — contributed 2 concept elements to this direction.
Getting Started with ADOL-C (2009) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_higher_order_automatic_differentiation_7bec8c8c,
  title = {higher-order automatic differentiation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/higher-order-automatic-differentiation/rdr_7bec8c8c544343e08bf4bbdad24e3068},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_higher_order_automatic_differentiation_7bec8c8c_v1,
  title = {higher-order automatic differentiation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/higher-order-automatic-differentiation/rdr_7bec8c8c544343e08bf4bbdad24e3068/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake