Low-rank surrogate modeling and stochastic zero-order optimization for training of neural networks with black-box layers

Andrei Chertkov, Artem Basharin, Mikhail Saygin, Evgeny Frolov, Stanislav Straupe, Ivan Oseledets

Here is a blog-style summary of the research paper.

TL;DR

The paper introduces astralora (Adaptive Surrogate TRAining with LOw RAnk), a novel framework designed to train hybrid neural networks that integrate non-differentiable physical layers (like photonic or neuromorphic hardware). By combining stochastic zeroth-order optimization with a dynamically updated low-rank surrogate model, the framework enables efficient end-to-end training without needing internal gradients or pre-calibrated "digital twins."

What Problem Does This Paper Address?

As conventional digital hardware struggles to keep up with the computational and energy demands of modern deep learning, alternative computing platforms—particularly linear photonic systems—have emerged as promising solutions for high-speed, energy-efficient inference.

However, training these hybrid digital-physical networks is notoriously difficult. Physical devices are inherently non-differentiable black boxes: they suffer from noise and drift, lack precise programmability, and cannot provide the internal gradient information required for standard backpropagation. Previous solutions either relied on rigid, offline-calibrated "digital twins" (which fail to adapt to real-time hardware drift) or full-model zeroth-order optimization (which scales poorly). The challenge is: How do we efficiently propagate gradients through a non-differentiable physical layer to train the rest of the network end-to-end?

Methods

The authors propose a two-part solution called astralora, which isolates the black-box (BB) physical layer and handles it using two distinct digital loops:

Figure 1: Schematic representation of the proposed method astralora. The diagram distin- guishes between the physical BB layer (central block), the digital surrogate optimization loop (bottom path), and the standard digital backpropagation flow (central path).
  1. Stochastic Zeroth-Order (ZO) Optimization: Since gradients cannot be extracted directly from the physical layer, the framework uses a stochastic finite-difference method (a Monte Carlo approach) to estimate the gradients needed to update the physical layer's internal parameters ($\omega$).
  2. Dynamic Low-Rank Surrogate Model (SM): To allow gradients to flow backward to the upstream digital layers, the framework approximates the physical layer's transfer matrix using a lightweight, rank-$r$ decomposition ($A_\omega \approx USV^T$). Because this surrogate model is fully digital and differentiable, standard automatic differentiation can pass right through it.
  3. Implicit Projector-Splitting Integrator (I-PSI): The core algorithmic innovation is the I-PSI scheme. Instead of performing a costly full matrix reconstruction or Singular Value Decomposition (SVD) after every training step, I-PSI efficiently updates the existing low-rank surrogate model by tracking only the incremental changes ($\Delta A$) between parameter updates. This reduces the required hardware queries from $O(d_{out} \cdot d_{inp})$ to a fixed budget independent of matrix dimensions: $N_{queries} = 2r + 2M_{sm}$.

Datasets

The framework was validated across three distinct domains using standard benchmarks:

  • Image Classification: CIFAR-10 dataset, evaluated using a deep convolutional neural network (CNN) with residual connections.
  • Audio Classification: UrbanSound8K dataset, evaluated using the ECAPA-TDNN architecture.
  • Language Modeling: A subset of the FineWeb corpus, evaluated using a large-scale, decoder-only GPT-2-like architecture (417M parameters).

Key Insights and Ideas

  • Perfect Backpropagation is Overrated: The most striking takeaway is that successful end-to-end training of hybrid systems does not require exact gradient calculations through physical layers. An approximate, dynamically updated low-rank surrogate is entirely sufficient.
  • Exploiting Low-Dimensional Subspaces: The method succeeds because gradient flow in deep neural networks naturally tends to reside in low-dimensional subspaces. Therefore, a very low-rank surrogate ($r \ll \min(d_{inp}, d_{out})$) can successfully capture the dominant input-output dynamics of the physical hardware.
  • Decoupled Efficiency: By separating parameter updates (via ZO) from gradient propagation (via the low-rank surrogate), the framework maintains the efficiency of standard backpropagation for the digital parts of the network while keeping hardware queries to a minimum.

Key Results

The authors emulated five distinct photonic architectures (including microring resonators, spatial light modulators, and Mach-Zehnder interferometers) and achieved near-parity with fully digital baselines:

  • CIFAR-10: Hybrid models achieved near-digital baseline accuracy when using a surrogate rank of $r \ge 10$ and a modest query budget ($M \ge 100$).
  • Audio Classification: On UrbanSound8K, all configurations achieved weighted F1-scores within 2% of the digital baseline ($77.46\%$) at rank $r=50$. The framework also proved robust against hardware nonlinearities, maintaining stable performance even under aggressive 3-bit and 4-bit weight quantization.
  • Language Modeling: In a massive stress test on a 417M-parameter GPT-2 model, the authors replaced up to 12 entire MLP blocks with black-box equivalents. The network remained highly trainable, showing only a graceful, predictable degradation in validation perplexity (moving from a baseline of 23.80 to ~30.87).

Limitations and Future Work

  • Simulation Bottlenecks: While the framework is designed to eliminate bottlenecks on real hardware, digitally emulating complex physical layers (like Mach-Zehnder interferometer meshes) is computationally expensive, making large-scale simulations slow.
  • Error Accumulation: As the number of replaced black-box layers increases, the small approximation errors introduced by each surrogate model accumulate, leading to slight performance drops in deep architectures.
  • Real Hardware Deployment: The current study relies entirely on highly accurate digital emulations of physical photonic devices. Future work will need to validate astralora on actual physical chips to prove its resilience against real-world environmental noise and hardware degradation.

Summary updated

Abstract

The growing demand for energy-efficient, high-performance AI systems has led to increased attention on alternative computing platforms (e.g., photonic, neuromorphic) due to their potential to accelerate learning and inference. However, integrating such physical components into deep learning pipelines remains challenging, as physical devices often offer limited expressiveness, and their non-differentiable nature renders on-device backpropagation difficult or infeasible. This motivates the development of hybrid architectures that combine digital neural networks with reconfigurable physical layers, which effectively behave as black boxes. In this work, we present a framework for the end-to-end training of such hybrid networks. This framework integrates stochastic zeroth-order optimization for updating the physical layer's internal parameters with a dynamic low-rank surrogate model that enables gradient propagation through the physical layer. A key component of our approach is the implicit projector-splitting integrator algorithm, which updates the lightweight surrogate model after each forward pass with minimal hardware queries, thereby avoiding costly full matrix reconstruction. We demonstrate our method across diverse deep learning tasks, including: computer vision, audio classification, and language modeling. Notably, across all modalities, the proposed approach achieves near-digital baseline accuracy and consistently enables effective end-to-end training of hybrid models incorporating various non-differentiable physical components (spatial light modulators, microring resonators, and Mach-Zehnder interferometers). This work bridges hardware-aware deep learning and gradient-free optimization, thereby offering a practical pathway for integrating non-differentiable physical components into scalable, end-to-end trainable AI systems.
BibTeX
@article{chertkov2026low,
  author = {Andrei Chertkov and Artem Basharin and Mikhail Saygin and Evgeny Frolov and Stanislav Straupe and Ivan Oseledets},
  title = {{Low-rank surrogate modeling and stochastic zero-order optimization for training of neural networks with black-box layers}},
  journal = {Neurocomputing},
  year = {2026},
  doi = {10.1016/j.neucom.2026.133808},
}
Figure