Sample-efficient Surrogate Model for Frequency Response of Linear PDEs using Self-Attentive Complex Polynomials

Andrew Cohen, Weiping Dou, Jiang Zhu, Slawomir Koziel, Peter Renner, Jan-Ove Mattsson, Xiaomeng Yang, Beidi Chen, Kevin Stone, Yuandong Tian

This paper introduces the Constant-Zeros-Poles (CZP) framework, a physics-informed surrogate model for linear Partial Differential Equations (PDEs) that predicts frequency responses as rational functions of complex polynomials. By combining this mathematically proven parametric form with a novel image-based representation and a Transformer architecture, the authors achieve significant gains in sample efficiency and optimization success for industrial antenna design.

TL;DR

The CZP framework leverages the mathematical structure of linear PDEs to predict frequency responses as ratios of complex polynomials, significantly outperforming standard neural networks. When applied to antenna design, it improves test loss by up to 25% and increases the success rate of reinforcement learning-based design optimization by 33%.

What Problem Does This Paper Address?

Designing modern technology—from antennas to medical imaging devices—relies on solving linear PDEs like Maxwell’s equations or the wave equation. However, two major bottlenecks exist:

  1. Computational Cost: High-fidelity simulations (e.g., Finite Element Methods) are extremely slow, taking anywhere from seconds to weeks for a single design iteration.
  2. Non-linearity: The relationship between a system's physical configuration (geometry) and its frequency response is highly non-linear and difficult to model.

While deep learning surrogate models can speed up this process, they often lack physical grounding, requiring massive amounts of expensive simulation data to generalize. They are also prone to producing "noisy" or non-physical predictions that can mislead optimization algorithms like Reinforcement Learning (RL).

Methods

The authors propose a two-pronged approach: a new mathematical framework and a specialized neural architecture.

The CZP Framework

The core contribution is a proof that for any discretized linear PDE, the temporal Fourier representation of the system's state can be written as a rational function of complex polynomials with respect to frequency $\omega$. For antenna design, this means the scattering coefficient $S_{11}(\omega)$ (which measures how much power is reflected back) follows a specific parametric form: $$\log |S_{11}(\omega)| = \log |c_0(h)| + \sum_{k=1}^{K} \log \frac{|\omega - z_k(h)|}{|\omega - p_k(h)|}$$ where $c_0$ is a constant, and $z_k$ and $p_k$ are the "zeros" and "poles" of the system, all of which are functions of the design vector $h$.

Image-Based Representation and Architecture

To capture the geometry of the antenna, the authors move beyond simple binary images. They propose a three-channel image representation (Figure 2) inspired by adaptive meshing:

  • Channel 1 & 2: Floating-point values representing the distance to the nearest boundary in the $x$ and $y$ directions.
  • Channel 3: A binary interior mask.

This representation is processed by a Visual Transformer (ViT). A CNN extracts local features, which are then tokenized and passed through a Transformer encoder to capture global spatial relationships. Finally, an MLP predicts the complex-valued constants, zeros, and poles.

Figure 2. Three channel image representation. Top: Boundary values represent distance to nearest pixel in the x-direction. Mid- dle: Boundary values represent distance to nearest pixel in the y-direction. Bottom: Binary interior of the antenna. This channel does not contain boundaries.

Datasets

The model was evaluated on a Five Patch Antenna design task.

  • Geometry: Five metallic patches on an FR-4 substrate (30mm x 6mm). The patches can move, creating a complex design space.
  • Data Generation: 48,000 samples were simulated using commercial software (CST Microwave Studio), with each simulation taking 90–120 seconds.
  • Baselines: The model was compared against standard MLPs, CNNs, and the Fourier Neural Operator (FNO).

Key Insights and Ideas

  • Physics as an Inductive Bias: The CZP form isn't just a heuristic; it's a proven property of linear PDEs. By forcing the neural network to predict the parameters of this form rather than the raw frequency values, the model is constrained to physically plausible, smooth outputs.
  • Smoothness Matters for Optimization: Standard "raw" prediction models often produce jagged frequency responses for unseen designs. During RL optimization, the agent can exploit these non-physical artifacts. Because CZP is smooth by design, it is much more robust to such "adversarial" exploitation (see Figure 8).
  • Boundary Awareness: The 3-channel image representation explicitly provides the network with boundary and corner information, which are the most critical regions for electromagnetic resonance, reducing the need for the model to "learn" these basic spatial features from scratch.

Key Results

  • Accuracy: CZP improved test loss by 10% to 25% compared to predicting raw frequency responses using the same architecture.
  • Sample Efficiency: The performance gap between CZP and baselines was most pronounced when training data was limited (e.g., using only 25% of the dataset).
  • Optimization Success: When used as a surrogate for RL (Soft Actor-Critic), CZP achieved a 33% higher success rate in finding antenna designs that met specific WiFi 6E frequency targets compared to raw prediction baselines.
Figure 6. Two successful antenna configurations (top row) and corresponding frequency responses (bottom row) predicted by the model (red) and computed by CST (green) found by optimization via RL for (a) CZP and (b) Raw.
Figure 6 shows how CZP (a) produces smooth, accurate predictions that match simulation (green) closely, whereas Raw prediction (b) can be non-smooth and less accurate.

Limitations and Future Work

  • Linearity Constraint: The CZP proof currently only applies to linear PDEs. Extending this to non-linear systems (like high-speed fluid dynamics) remains an open challenge.
  • Geometric Complexity: The study focused on 2D patch antennas. The authors note that future work is needed to scale the approach to more complex 3D structures.
  • Tokenization Efficiency: While effective, image-based representations can be computationally heavy. The authors suggest exploring more efficient tokenization schemes that retain the same level of spatial information.

Summary updated

Abstract

Linear Partial Differential Equations (PDEs) govern the spatial-temporal dynamics of physical systems that are essential to building modern technology. When working with linear PDEs, designing a physical system for a specific outcome is difficult and costly due to slow and expensive explicit simulation of PDEs and the highly nonlinear relationship between a system's configuration and its behavior. In this work, we prove a parametric form that certain physical quantities in the Fourier domain must obey in linear PDEs, named the CZP (Constant-Zeros-Poles) framework. Applying CZP to antenna design, an industrial application using linear PDEs (i.e., Maxwell's equations), we derive a sample-efficient parametric surrogate model that directly predicts its scattering coefficients without explicit numerical PDE simulation. Combined with a novel image-based antenna representation and an attention-based neural network architecture, CZP outperforms baselines by 10% to 25% in terms of test loss and also is able to find 2D antenna designs verifiable by commercial software with $33\%$ greater success than baselines, when coupled with sequential search techniques like reinforcement learning.
BibTeX
@misc{cohen2023sample,
  author = {Andrew Cohen and Weiping Dou and Jiang Zhu and Slawomir Koziel and Peter Renner and Jan-Ove Mattsson and Xiaomeng Yang and Beidi Chen and Kevin Stone and Yuandong Tian},
  title = {{Sample-efficient Surrogate Model for Frequency Response of Linear PDEs using Self-Attentive Complex Polynomials}},
  year = {2023},
  eprint = {2301.02747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.02747},
}
Figure