In Situ Training of Optical Neural Networks by Coding Agents

Ilker Oguz, Colin Christian, Christophe Moser, Demetri Psaltis, Gordon WetzsteinarXiv 2026 · arXiv

This paper presents a fascinating intersection of Large Language Models (LLMs) and physical hardware optimization. Researchers from Stanford and EPFL have demonstrated that "coding agents"—LLMs capable of writing and executing code—can autonomously train Optical Neural Networks (ONNs) directly on hardware, bypassing the need for expert-level manual calibration.

TL;DR

The researchers introduced "optics as code," a framework where an LLM-based agent trains optical neural networks in situ by interacting with hardware through a software harness. The agent develops an emergent physical understanding of the system, outperforming traditional gradient-free optimization and matching the accuracy of expert-calibrated digital twins at a fraction of the cost.

What Problem Does This Paper Address?

Optical Neural Networks (ONNs) promise massive speed and energy efficiency by processing information with light. However, they are notoriously difficult to deploy. Every physical device has unique imperfections—fabrication errors, misalignments, and aberrations—meaning a model trained in a "perfect" simulation rarely works on real hardware.

To fix this, researchers typically use two methods:

  1. Model Calibration: Building a "digital twin" of the specific device. This requires an expert physicist, thousands of measurements, and heavy computation.
  2. Gradient-Free Optimization: Trial-and-error methods like Reinforcement Learning. These are slow and scale poorly as the number of parameters increases.

The authors sought a way to train these networks on-device without needing an expert to build a custom model for every single unit.

Methods

The team developed a method called optics as code. Instead of a human physicist, a coding agent (using models like Claude 3.5 Opus) is placed in an isolated workspace.

  • The Harness: The agent interacts with the optical hardware through a "harness"—a software interface. To make a measurement, the agent must submit a hypothesis and a snippet of code. The harness then executes the code on the Spatial Light Modulator (SLM), captures camera images of the result, and returns them to the agent.
  • Physical Reasoning: Rather than just tweaking raw pixels, the agent writes programs using optical primitives—lenses, prisms, and Zernike polynomials. It runs "probes" to measure how the system responds to tilt or focus, effectively performing its own system identification.
  • Knowledge Transfer: The agent can record its findings in a "notes" file. In subsequent sessions, it can read these notes to understand the specific quirks of the hardware (e.g., "the beam is slightly astigmatic"), allowing it to skip the discovery phase and go straight to optimization.
The "optics as code" framework compared to traditional in situ training methods.

Datasets

The researchers evaluated the agent on two standard computer vision benchmarks adapted for optical hardware:

  • MNIST: A subset of handwritten digits (0, 1, and 2) for a classification task.
  • Fashion-MNIST: Used for an image encoding task, where the ONN compresses images into a latent representation that a digital decoder then reconstructs.

Key Insights and Ideas

  • Emergent Experimentalist Behavior: The most striking finding is that the agent acts like a human physicist. It doesn't just guess; it designs experiments to find the "center" of the optical planes and measures how much a phase ramp displaces a spot on the camera.
  • Interpretable Reparametrization: The agent discovered that it could replace hundreds of thousands of individual pixel updates with a few physically interpretable parameters (like focal length or tilt). This "compresses" the optimization problem, making it much more efficient.
  • Scaling with Intelligence: The performance of the optical system improved as the researchers used more capable LLMs. This suggests that as AI models get smarter, our ability to calibrate complex physical experiments will grow automatically.
  • The Power of Notes: Providing the agent with notes from a previous session (even without the previous phase masks) allowed it to reach high accuracy much faster, demonstrating that physical "knowledge" of a device is highly transferable.
Reasoning traces showing the agent's step-by-step discovery of the system's physical properties.

Key Results

  • Superior Accuracy: On the MNIST classification task, the agent reached 96.8% accuracy. This crushed gradient-free baselines like PPO (~52.5%) and even slightly outperformed backpropagation through a meticulously calibrated expert model (96.3%).
  • Data Efficiency: The agent achieved these results using significantly fewer measurements than are typically required to even calibrate a digital twin (which took 67,000 images in this study).
  • Image Encoding: In the Fashion-MNIST task, the agent achieved a reconstruction PSNR of 16.1 dB, outperforming the 15.0 dB reached by standard reinforcement learning methods.
Comparison of classification accuracy across different training methods.

Limitations and Future Work

While impressive, the authors note several areas for growth:

  • Measurement Budget: The agent's performance is still bound by the number of physical measurements allowed. Finding the optimal balance between "thinking" (LLM reasoning) and "doing" (hardware measurements) is an open question.
  • Complexity Scaling: This study focused on a system equivalent to a few layers. Scaling this to deep, many-layered optical networks will require the agent to manage even more complex inter-layer dependencies.
  • Speed: While the ONN itself runs at the speed of light, the training loop is limited by the inference speed of the LLM. However, for one-time calibration of mass-produced devices, this is a highly acceptable trade-off.

Ultimately, "optics as code" suggests a future where complex scientific instruments don't need manual tuning by PhDs—they might just need a smart enough agent and a well-designed harness.

Summary updated

Abstract

Optical neural networks promise fast, energy-efficient computing, but every device departs from its design and must be trained with the hardware. In situ training relies on an expert-built calibrated digital model of each device or on gradient-free trial and error. Coding agents, large language models that write and run programs, have shown an emergent ability to understand and edit the weights of digital neural networks. Here we introduce optics as code, a method in which a coding agent trains an optical neural network from measurements alone, through a harness that returns camera images of its probes and requires a hypothesis per request. We discovered that the agent forms a physical understanding of the optics and reparametrizes the network with interpretable optical elements tuned on the hardware. It outperforms gradient-free methods in classification and image encoding, approaching backpropagation through a calibrated model at a fraction of the cost. Needing neither per-device calibration nor gradients through the hardware, and improving with more capable models, optics as code offers a practical route to large, scalable optical neural networks and a general tool for calibrating and optimizing optical experiments.
BibTeX
@misc{oguz2026situ,
  author = {Ilker Oguz and Colin Christian and Christophe Moser and Demetri Psaltis and Gordon Wetzstein},
  title = {{In Situ Training of Optical Neural Networks by Coding Agents}},
  year = {2026},
  eprint = {2611.23875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23875},
}
Figure