Conditional Dynamical Systems for Image Generation

Lianlong Sun, Chuan Liu, Tong Geng, Michael HuangarXiv 2026 · arXiv

TL;DR

This paper introduces a novel framework for conditional image generation designed to run natively on non-von Neumann hardware by framing generation as an energy-descent process. By introducing "energy tilting"—where class conditioning is achieved via a simple linear field rather than reprogramming expensive hardware connections—the authors achieve high-quality image generation (9.71 FID on CIFAR-10) while significantly reducing hardware complexity.

What Problem Does This Paper Address?

Modern image generation relies heavily on deep generative models running on power-hungry GPU infrastructures, raising long-term sustainability and efficiency concerns. Emerging non-von Neumann computing platforms (such as quantum annealers, Ising machines, and photonic processors) offer a promising alternative due to their extreme energy efficiency.

However, most existing work simply forces these unique physical substrates to perform conventional neural network operations like matrix multiplication. This fails to exploit their most powerful native capability: the fact that physical systems naturally relax toward low-energy states at virtually zero computational cost. Furthermore, for conditional generation (e.g., creating a specific class of image like a "cat"), existing methods require rewriting the entire pairwise interaction network for each class, which is slow, expensive, and impractical on real hardware.

Methods

The authors propose a family of continuous dynamical systems where image generation is driven directly by energy minimization.

  1. Energy-Descending Dynamics: The generator's internal state $x(t)$ evolves continuously by descending an explicit scalar Lyapunov energy landscape ($\dot{x}(t) = -\nabla E(x(t))$). The system uses a continuous, Ising-inspired state bounded within $[-1, 1]^n$.
  2. Energy Tilting for Conditioning: To introduce class conditioning without rebuilding the hardware's physical connection fabric, the dense pairwise interaction matrix ($A$) is shared across all classes. Conditioning is achieved entirely by applying a class-specific linear external field ($d_y$) that "tilts" the energy landscape: $$E_y(x) = -\frac{1}{2}x^\top Ax - \gamma d_y^\top x$$
  3. Decoupled Architecture: The system evolves over a fixed time horizon $T$. The terminal state $x(T)$ is then passed to a compact, class-agnostic convolutional decoder that renders the internal representation into a final image.

The model is trained digitally using a "drifting-loss" framework, which optimizes the energy landscape based on the quality of the samples produced at the end of the finite time horizon.

Figure 1: Generation by energy descent, a computa- tion that can be performed natively at very low cost on emerging non-von Neumann hardware. Class con- ditioning tilts the energy landscape.

Datasets

The framework was trained and evaluated on the standard CIFAR-10 dataset. Performance was quantified using the clean-FID metric calculated across 50,000 class-balanced generated samples.

Key Insights and Ideas

  • Computation via Relaxation: Instead of treating physical relaxation as an implementation detail, this work elevates it to the core computational primitive. The trajectory of the energy descent is the generative computation.
  • The Power of Tilting: Class conditioning does not require altering network topology. A simple linear external field (biases) can shift the energy landscape enough to reliably guide the system toward states representing the target class.
  • Hardware Compatibility: By keeping the quadratic interaction matrix $A$ fixed and shared, the dense physical coupling fabric of an Ising machine or photonic chip can remain programmed and calibrated, while class switching only requires modifying a lightweight external vector.

Key Results

  • High-Quality Generation: At a state dimension of $n=4096$, the energy-tilting model (Ising-4096-ET) achieved a clean-FID of 9.71 on CIFAR-10, slightly outperforming the baseline model (9.92) while eliminating the need to reprogram the interaction matrix.
  • Efficiency Gains: As shown in the complexity analysis, energy tilting reduces the total parameter count across all tested dimensions because it strips away the auxiliary conditioning subsystems used in prior architectures.
Table 1: Model complexity and generation quality on CIFAR-10. Rows without a suffix use the symmetric-coupling baseline with an auxiliary conditioning subsystem; -ET rows use energy tilting, with a shared interaction matrix and one learned external field per class. “Reprogram Needed” indicates whether changing the target class requires changing the interaction array. Clean-FID is computed from 50K generated samples; lower is better.
  • Mechanism Verification: The authors empirically verified that the external fields reshape the landscape as intended. When evaluating terminal states under different class fields, the matched-class field consistently assigned significantly lower energy to the states than mismatched fields.
Verification of the energy-tilting mechanism across CIFAR-10 classes

Limitations and Future Work

  • Simulation vs. Physical Hardware: The primary limitation is that the dynamics were simulated digitally rather than deployed on physical analog hardware. Testing the model under real-world constraints like substrate noise, limited precision, and calibration errors remains a critical next step.
  • Dense Connectivity: The current interaction matrix $A$ is fully dense. Exploring sparse or structured connectivity will be necessary to fit within the physical routing limitations of many hardware chips.
  • Energy Function Expressivity: The model relies on a simple quadratic energy function to maintain compatibility with current Ising machines. Scaling to more complex datasets may require exploring richer, higher-order energy functions.

Summary updated

Abstract

Image generation has been dominated by deep generative models running on GPUs, a paradigm whose computational and energy costs raise growing sustainability concerns. Emerging non-von Neumann computing substrates, including quantum, compute-in-memory, photonic, and thermodynamic platforms, promise greater efficiency, yet much of the existing work ports conventional neural architectures onto them and primarily accelerates operations such as matrix multiplication. This does not fully exploit a native capability of many emerging computing substrates: relaxation toward low-energy states can itself perform computation at negligible cost. We develop a family of continuous dynamical systems for image generation, built around this primitive to better harness its computational power. The proposed generator evolves an internal state under dynamics admitting an explicit Lyapunov energy and then renders the resulting state through a compact, class-agnostic decoder. For conditional generation, we introduce energy tilting: programmed pairwise interactions remain fixed and shared across classes, while a class-dependent linear field reshapes the energy without reprogramming the interaction array. An Ising-inspired design reaches a clean-FID of 9.71 on CIFAR-10 with 4096 spin variables. These results suggest that the energy-descending dynamics can serve directly as a generative computation and offer a promising path toward efficient generative tasks beyond GPUs.
BibTeX
@misc{sun2026conditional,
  author = {Lianlong Sun and Chuan Liu and Tong Geng and Michael Huang},
  title = {{Conditional Dynamical Systems for Image Generation}},
  year = {2026},
  eprint = {2608.14961},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.14961},
}
Figure