Variational framework for partially-measured physical system control: examples of vision neuroscience and optical random media

Babak Rahmani, Demetri Psaltis, Christophe MoserarXiv 2021 · arXiv

Abstract

Controlling complex, nonlinear physical systems with only partial measurements presents a significant challenge in fields ranging from optics to neuroscience. We present a generative framework that utilizes Variational Auto-Encoders (VAEs) to jointly train a probabilistic forward model and an "Actor" network for inverse control. This approach successfully identifies precise inputs to achieve desired outputs in stochastic environments, demonstrating high performance in optical phase retrieval and neural stimulus optimization.

Introduction

In many scientific and engineering domains, the goal is to "control" a system by finding an input $x$ that produces a specific target output $y^*$. While linear systems can be managed using transmission matrices, real-world systems—such as light propagation through turbid media or neural responses in the brain—are often nonlinear, time-varying, and "partially measured." Partial measurement, such as capturing only light intensity while losing phase information, creates a "black-box" scenario where traditional analytical models and standard supervised learning often fail due to data scarcity or stochasticity.

Methodology: The Actor-Model Framework

The proposed solution involves a dual-network architecture designed to learn the system's physics implicitly while performing inverse mapping.

The dual-network architecture consists of a VAE-based Model (enc/dec) that learns the system's forward physics and an Actor network that predicts the necessary input for a target output.
  1. The Model (Forward Estimator): A Variational Auto-Encoder (VAE) is employed to learn the probability distribution of the system's behavior. By utilizing a latent space ($z$), the model accounts for noise and unmeasured variables, making it robust to the stochastic nature of physical experiments.
  2. The Actor (Backward Mapping): This network performs the inverse task, taking a desired target $y^*$ and predicting the required input $x^*$.
  3. Joint Iterative Learning: The Model and Actor are trained iteratively. The Model refines its understanding of the physics from experimental samples, while the Actor uses the Model's current state to improve its input predictions.

Experimental Validation

The framework was tested on two high-complexity datasets:

  • Optical Random Media: Controlling light through a 75cm multimode fiber to project MNIST handwritten digits. The system provided only intensity-only measurements, requiring the model to solve a complex phase retrieval problem.
  • Vision Neuroscience: Optimizing stimuli for Salamander Retina Ganglion Cells (RGCs). The objective was to find low-resolution images that elicit specific neural spike counts, matching the response of original high-resolution stimuli.

Results and Discussion

The framework demonstrated rapid convergence and high accuracy across both domains.

Training dynamics and latent space evolution. (a) and (b) show the decrease in loss and increase in Pearson correlation over iterations. The bottom panels visualize the convergence of the latent space embeddings.

In the optics task, the algorithm achieved a 2D Pearson correlation of approximately 0.9, comparable to full-measurement techniques that require more sophisticated hardware. In neuroscience, the model reached a correlation of ~0.3 within three iterations, which is near the theoretical maximum given the inherent biological noise of the RGCs.

A key insight is the behavior of the latent space. As shown in the visualizations, the model's internal representation ($z$) converges over iterations, aligning more closely with the true underlying physical state. This suggests that the generative nature of the VAE allows the Actor to effectively "infer" missing information, such as phase in optics, to achieve precise control.

Conclusion

This variational approach offers a robust method for controlling "unseen" physical processes. By prioritizing probabilistic modeling over deterministic mapping, the framework handles noise and partial measurements effectively. Future work will focus on scaling this approach to larger systems and incorporating explicit physical priors to further enhance performance in chaotic or rapidly changing environments.

Summary updated

Abstract

To characterize a physical system to behave as desired, either its underlying governing rules must be known a priori or the system itself be accurately measured. The complexity of full measurements of the system scales with its size. When exposed to real-world conditions, such as perturbations or time-varying settings, the system calibrated for a fixed working condition might require non-trivial re-calibration, a process that could be prohibitively expensive, inefficient and impractical for real-world use cases. In this work, we propose a learning procedure to obtain a desired target output from a physical system. We use Variational Auto-Encoders (VAE) to provide a generative model of the system function and use this model to obtain the required input of the system that produces the target output. We showcase the applicability of our method for two datasets in optical physics and neuroscience.
BibTeX
@misc{rahmani2021variational,
  author = {Babak Rahmani and Demetri Psaltis and Christophe Moser},
  title = {{Variational framework for partially-measured physical system control: examples of vision neuroscience and optical random media}},
  year = {2021},
  eprint = {2110.13228},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.13228},
}
Figure