Sample-efficient Surrogate Model for Frequency Response of Linear PDEs using Self-Attentive Complex Polynomials

Andrew S. Cohen, Weiping Dou, Jiang Zhu, Sławomir Kozieł, Peter Renner, Jan-Ove Mattsson, Xiaomeng Yang, Beidi Chen, Kevin H. Stone, Yuandong Tian

Abstract

Linear Partial Differential Equations (PDEs) govern the spatial-temporal dynamics of physical systems that are essential to building modern technology. When working with linear PDEs, designing a physical system for a specific outcome is difficult and costly due to slow and expensive explicit simulation of PDEs and the highly nonlinear relationship between a system's configuration and its behavior. In this work, we prove a parametric form that certain physical quantities in the Fourier domain must obey in linear PDEs, named the CZP (Constant-Zeros-Poles) framework. Applying CZP to antenna design, an industrial application using linear PDEs (i.e., Maxwell's equations), we derive a sample-efficient parametric surrogate model that directly predicts its scattering coefficients without explicit numerical PDE simulation. Combined with a novel image-based antenna representation and an attention-based neural network architecture, CZP outperforms baselines by 10% to 25% in terms of test loss and also is able to find 2D antenna designs verifiable by commercial software with 33% greater success than baselines, when coupled with sequential search techniques like reinforcement learning.

Introduction

The design of complex physical systems, such as antennas or heat sinks, is fundamentally constrained by the need to solve linear Partial Differential Equations (PDEs). Traditional numerical methods, while accurate, are computationally expensive and slow, creating a bottleneck in iterative design processes. While deep learning surrogate models have emerged as a faster alternative, they often lack physical grounding, treating frequency responses as arbitrary curves. This paper introduces the Constant-Zeros-Poles (CZP) framework, a physics-informed approach that leverages the mathematical structure of linear PDEs to create a sample-efficient surrogate model.

Methodology

The CZP framework integrates physical inductive biases through a parametric form and a specialized geometric representation.

The CZP Framework

The authors prove that the frequency response of a discretized linear PDE follows a specific rational function of complex polynomials. For electromagnetic scattering coefficients, this response is characterized by a global constant, zeros, and poles. By training a Vision Transformer (ViT) to predict these parameters rather than the raw response curve, the model ensures that its outputs are physically consistent.

Image-Based Antenna Representation

To effectively capture the geometric features that drive resonance, the authors developed a three-channel image representation. This approach encodes the distance to the nearest boundary in both x and y directions alongside a binary mask of the antenna's interior. This multi-channel input provides the network with explicit information about edges and corners, which are critical for accurate electromagnetic simulation.

The three-channel image representation capturing boundaries and interior.

Results and Discussion

The framework was validated using a dataset of 48,000 2D antenna designs. The CZP-based model demonstrated a 10% to 25% improvement in test loss compared to models predicting raw curves.

Physical Plausibility and Robustness

A significant advantage of the CZP framework is its inherent "smoothness." Because the model outputs physical parameters (zeros and poles), the reconstructed frequency response remains physically plausible even in cases where prediction error is higher. In contrast, "raw" prediction models often produce noisy artifacts that violate physical principles.

Comparison of CZP (smooth by design) vs. Raw prediction (noisy) on challenging test examples.

Design Optimization Success

When utilized within a Reinforcement Learning (RL) loop for antenna design, the CZP surrogate model achieved a 33% higher success rate in finding valid designs compared to standard deep learning baselines. This highlights the framework's effectiveness in practical engineering optimization tasks.

Conclusion

The CZP framework offers a robust, physics-informed method for accelerating the design of systems governed by linear PDEs. By combining a parametric form derived from PDE theory with a boundary-aware image representation, the authors achieve superior accuracy and physical consistency. Future research will explore extending this framework to 3D geometries and non-linear systems.

Summary updated

Abstract

Linear Partial Differential Equations (PDEs) govern the spatial-temporal dynamics of physical systems that are essential to building modern technology. When working with linear PDEs, designing a physical system for a specific outcome is difficult and costly due to slow and expensive explicit simulation of PDEs and the highly nonlinear relationship between a system's configuration and its behavior. In this work, we prove a parametric form that certain physical quantities in the Fourier domain must obey in linear PDEs, named the CZP (Constant-Zeros-Poles) framework. Applying CZP to antenna design, an industrial application using linear PDEs (i.e., Maxwell's equations), we derive a sample-efficient parametric surrogate model that directly predicts its scattering coefficients without explicit numerical PDE simulation. Combined with a novel image-based antenna representation and an attention-based neural network architecture, CZP outperforms baselines by 10% to 25% in terms of test loss and also is able to find 2D antenna designs verifiable by commercial software with $33\%$ greater success than baselines, when coupled with sequential search techniques like reinforcement learning.
BibTeX
@misc{cohen2023sample,
  author = {Andrew S. Cohen and Weiping Dou and Jiang Zhu and Sławomir Kozieł and Peter Renner and Jan-Ove Mattsson and Xiaomeng Yang and Beidi Chen and Kevin H. Stone and Yuandong Tian},
  title = {{Sample-efficient Surrogate Model for Frequency Response of Linear PDEs using Self-Attentive Complex Polynomials}},
  year = {2023},
  eprint = {2301.02747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.02747},
}
Figure