In-Context Operator Learning for Partial Differential Equations

30 concepts24 papers1 proposal

In scientific computing, solving partial differential equations (PDEs) has traditionally been an instance-specific endeavor. Classical numerical solvers and early physics-informed neural networks (PINNs) compute solutions for one specific set of boundary conditions, initial states, or equations at a time. If any of these parameters change, the entire computational process must start from scratch.

As surveyed by Joe Watson et al. in their overview of physics-informed machine learning (Machine Learning with Physics Knowledge for Prediction: A Survey), and argued by Madison Cooley et al. (HyResPINNs: A Hybrid Residual Physics-Informed Neural Network Architecture Designed to Balance Expressiveness and Trainability), overcoming this limitation requires shifting from solving individual instances to learning the underlying solution operator—the mathematical mapping between entire function spaces. By approximating the full operator, a single trained model can instantly generalize across varying conditions without instance-specific retraining (Jae Yong Lee in Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learning; Robert W. Clayton et al. in Ambient Noise Full Waveform Inversion with Neural Operators).

The frontier of this field has moved toward in-context operator learning. Borrowing the core philosophy behind Large Language Models (LLMs), this paradigm trains foundation models to adapt to entirely new physical systems, geometries, and PDE types at inference time without updating any network weights.


The Intuition: Prompting the Laws of Physics

How can a neural network solve an unseen differential equation without changing its weights? The key lies in treating the physical context as a prompt.

During a large-scale pre-training phase, a model is exposed to a massive, diverse library of distinct differential equations. Instead of memorizing a specific physical law, the network learns a shared mathematical prior over how fields evolve, how boundaries constrain solutions, and how coefficients modulate dynamics.

At inference time, the model is provided with a "context prompt" that grounds it in a specific physical regime. This prompt can take several forms:

1. Data Prompts (Input-Output Pairs)

Liu Yang et al. introduced this concept with In-Context Operator Networks (ICON) in In-context operator learning with data prompts for differential equation problems. By feeding the network a few local input-output observations of an unknown system, the model infers the underlying operator on the fly. As Wuyang Chen demonstrates in Automated, Interpretable, and Scalable Scientific Machine Learning, this conditioning allows the model to handle highly complex, out-of-distribution physical regimes—such as chaotic or high-frequency dynamics—without the computational overhead of retraining.

2. Symbolic and Conditional Prompts

Rather than forcing the model to guess the equation from data pairs, we can explicitly pass the mathematical structure itself. Hang Zhou and Yuezhou Ma developed this approach in Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solvers by conditioning a Transformer backbone on a complete set of mathematical components, including symbolic representations, coefficients, and boundary conditions.

Similarly, the authors of PDEformer: Towards a Foundation Model for One-Dimensional Partial Differential Equations (Zhanhong Ye et al.) represent equations as computational graphs. By learning the broad "language of physics," these models achieve zero-shot generalization across entirely different equation families, matching or outperforming specialized expert models trained from scratch.

[Symbolic Graph / Data Pairs] ---> [Pre-trained Transformer] ---> [Instant Zero-Shot Solution]

Scaling Up to Multi-Dimensional Foundation Models

To capture universal physical features that transfer across dimensions and scales, neural operators are scaling up in both parameter size and architectural flexibility.

  • Scaling Parameters: Zhongkai Hao et al. developed DPOT (DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training), scaling a Fourier-based transformer up to 1 billion parameters. By utilizing an auto-regressive denoising pre-training strategy, the model exhibits strong zero-shot generalization and high data efficiency when transferred to novel physical simulations, including complex 3D systems.
  • Cross-Dimensional Unification: The authors of OmniArch: Building Foundation Model for Scientific Computing (Tianyu Chen et al.) created a unified architecture capable of processing 1D, 2D, and 3D spatial dimensions simultaneously. By combining Fourier spatial encodings with a Transformer backbone, OmniArch exhibits emergent in-context learning: providing the model with a few initial observation steps as a prompt allows it to instantly adapt its predictions to novel PDEs.

Achieving Geometric and Mesh Agnosticism

A major hurdle for classical simulators is geometric complexity; a model trained on a square grid typically fails when applied to an airfoil or an urban landscape. In-context operator learning bypasses this by capturing the fundamental properties of differential operators independently of the underlying spatial discretization.

Generalizing Across Complex Domains

Using the massive FlowBench dataset, Ronak Tali et al. demonstrated in FlowBench: A Large Scale Benchmark for Flow Simulation over Complex Geometries that broad pre-training allows foundation models to capture universal fluid features that seamlessly transfer to entirely unseen, highly complex geometric domains.

This mesh-agnostic behavior is also evident in linear system acceleration. Yichen Li et al. (Learning Preconditioners for Conjugate Gradient PDE Solvers) showed that neural preconditioners trained on simple geometric meshes generalize effectively to complex engineering domains without retraining. Similarly, Niklas Freymuth et al. (Physics-informed MeshGraphNets (PI-MGNs): Neural finite element solvers for non-stationary and nonlinear simulations on arbitrary meshes) proved that graph-based neural solvers trained on small, generic meshes scale seamlessly to much larger, unseen meshes.

Decoupling Physics from Spatial Grids

To achieve true zero-shot generalization, models must learn the mapping of the function space rather than memorizing spatial patterns:


Self-Supervision and Physics-Informed Tuning

Generating high-fidelity simulation data for pre-training is computationally expensive. To mitigate this, modern operator learning increasingly embeds physical laws directly into the self-supervision and fine-tuning loops.

As See-Kiong Ng notes in PIED: Physics-Informed Experimental Design for Inverse Problems, transitioning toward operator architectures is essential for achieving zero-shot capabilities across diverse physical systems. By training directly on the PDE operator itself rather than pre-computed datasets, models can leverage physics-based self-supervision to generalize to novel initial conditions (Nathan Gaby et al. in Neural Control of Parametric Solutions for High-dimensional Evolution PDEs).

Furthermore, incorporating physical laws during downstream adaptation allows for rapid, data-free specialization. The authors of DeepONet as a multi-Operator extrapolation model: Distributed pretraining with physics-Informed fine-Tuning (Zecheng Zhang et al.) demonstrate that combining distributed pre-training with physics-informed fine-tuning allows operators to adapt to entirely new tasks without requiring explicit downstream ground-truth data.

Once this operator mapping is learned, inference becomes incredibly fast. For example, Sourav Pal et al. (PINNs with Learnable Quadrature) developed a framework that predicts optimal quadrature rules, allowing the model to solve entirely new instances of a PDE family—complete with unseen forcing functions—in a single forward pass.

By combining the flexible prompting mechanics of foundation models with the rigid constraints of mathematical physics, in-context operator learning provides a path toward universal, real-time neural solvers that adapt to the physical world as dynamically as language models adapt to text.

Key Foundational Work

In-context operator learning with data prompts for differential equation problems (2023) — contributed 3 concept elements to this direction.
Pde Generalization of In-Context Operator Networks: A Study on 1d Scalar Nonlinear Conservation Laws (2024) — contributed 3 concept elements to this direction.
Can we pre-train ICL-based SFMs for the zero-shot inference of the 1D CDR problem with noisy data? (2024) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_in_context_operator_learning_436f861a,
  title = {in-context operator learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/in-context-operator-learning/rdr_436f861a8adc489bbfef8907dae7ae40},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_in_context_operator_learning_436f861a_v1,
  title = {in-context operator learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/in-context-operator-learning/rdr_436f861a8adc489bbfef8907dae7ae40/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake