Global Context from Small Chebyshev PDE Solves Inside Vision Networks

Convolutional networks see only a small neighborhood per layer. The popular fix in frequency-domain vision models is to take a Fourier transform of each feature map and multiply it by a learned filter, which amounts to one image-wide convolution at $O(N \log N)$ cost. We propose a different way to reach every pixel in one layer: have the layer solve a small differential equation. Each channel's output is the solution $u$ of a screened-Poisson-type equation whose right-hand side is the input feature map and whose coefficients are learned smooth functions of position. The solve is done with a spectral method, meaning the solution is written as a short sum of smooth global basis functions (here Chebyshev polynomials, which live on a bounded interval and need no wrap-around).

The solution of such an equation depends on the input at every location, so the receptive field is global. Its decay length is set by learned coefficients, it varies with absolute position, and it respects image borders instead of wrapping left edge onto right. In a Chebyshev basis with polynomial coefficients these equations become almost-banded linear systems, which can be solved in linear time and without approximation error. The only approximation is truncating the input to $K$ modes. Because the solve smooths its input by two derivatives, that truncation costs much less on $u$ than on the raw features.

Core Claim

The claim is that a global mixer built from an exactly discretized, position-dependent elliptic operator beats diagonal Fourier filters on dense prediction and on changes of resolution. It should also be cheaper than decomposed attention. Its advantage should come specifically from position-varying coefficients, not from avoiding periodic boundaries. We make that distinction because a skeptic can remove wrap-around with zero padding, and we test that alternative directly.

"Spectral accuracy" here means something precise and modest. Natural feature maps have edges, so their expansions converge only algebraically, and we do not claim exponential convergence for them. What we do claim has two parts. First, the operator is represented exactly: polynomial-coefficient differential operators are banded in ultraspherical bases, following A Sparse Spectral Method on Triangles, which solves such systems to $3\times10^{-16}$ error. Second, the output's coefficients decay faster than the input's by roughly $k^{-2}$. That factor comes from inverting a second-order operator, since mode $k$ of the right-hand side is divided by about $1+\ell^2 k^2$.

Two consequences follow and can be tested. The global branch can run on very few modes. The same learned operator can be applied at any resolution.

Mechanism

Take one row of a feature map, rescaled to $x\in[-1,1]$. Project it onto $K$ Chebyshev modes with a precomputed $n\times K$ least-squares matrix. Then solve

$$\big(c(x) - \partial_x\, a(x)\, \partial_x\big)\, u = f, \qquad \alpha u \pm \beta \partial_x u = 0 \ \text{at } x=\pm 1,$$

where $f$ is the projected input row and $u$ is the output row. The coefficients $a(x)>0$ and $c(x)>0$ are Chebyshev polynomials of low degree $d$. Their ratio sets a local decay length $\ell(x)=\sqrt{a/c}$, and $\alpha,\beta$ are learned boundary weights of Robin type, which interpolate between fixed-value and fixed-slope conditions at the border. In an ultraspherical basis this is a banded matrix with bandwidth about $d+2$, plus two dense boundary rows, so it solves in $O(Kd^2)$ per row. Gradients come from one more banded solve with the transpose.

We do this along rows and then along columns, as an alternating-direction pair. Each 1D solution depends on the whole line, so the pair couples every pixel to every other pixel. This is the same two-hop routing as CCNet: Criss-Cross Attention for Semantic Segmentation, but it costs $O(NK)$ rather than $O(N\sqrt N)$.

The polynomial coefficients are set per channel group. A small network on globally pooled features predicts them, which makes the operator content-adaptive. A parallel local $3\times3$ branch carries edges, borrowing the split used by Fast Fourier Convolution. Unlike FFC's fixed channel ratio, the division of labor here has a stated reason: the global branch only ever produces a field smoothed by the operator, which is where spectral truncation is efficient.

Relation to Prior Work

Three vision papers use the Fourier transform for global mixing. Fast Fourier Convolution applies $1\times1$ convolutions to FFT coefficients. Global Filter Networks for Image Classification multiplies them by a learned complex filter the size of the grid. Butterworth as Attention parameterizes that filter as an anisotropic Butterworth gate and names Chebyshev filter families as future work.

All three share two properties. Their operators are diagonal in frequency, which makes them shift-invariant up to cyclic shifts. Their parameters, at least in GFNet, are tied to one grid size. Our operator is instead not diagonal in any basis, depends on absolute position through $a(x)$ and $c(x)$, and has a parameter count set by $d$ rather than by resolution. These papers motivate the proposal and supply the local/global split. They are also the main competitors.

Neural Spectral Methods predicts Chebyshev coefficients of PDE solutions and computes derivatives algebraically. SNODE represents 1D trajectories with Legendre series. We reuse their machinery of algebraic derivatives on coefficients, but use the PDE as a layer inside a vision network rather than as the target to be learned.

ParCNetV2 constrains the design. It shows that breaking shift invariance through border effects helps, reaching 84.6% ImageNet top-1 for ParCNetV2-B. Position awareness is therefore valuable, and our design has to earn it more cheaply than oversized kernels do.

The vision-transformer pressure surrogate in Efficient Vision Transformer-based Surrogate for Scalable Pressure Prediction in Incompressible Turbulent Flows uses attention to approximate an elliptic, Poisson-type solve. We build that solve into the architecture directly.

Research Program

We will build the layer in PyTorch with batched banded solvers. It will be inserted into two backbones. The first is a ResNet-50, following FFC's substitution protocol. The second is a GFNet-XS-sized isotropic model, where it replaces the global filter layer.

The analysis has three parts. The first measures coefficient decay of $f$ and $u$ on trained networks, to check the $k^{-2}$ gap. The second visualizes learned decay lengths $\ell(x)$, to see whether they vary with position, for instance shorter near borders. The third checks resolution consistency: we apply a 224-trained model at 384 and 512 pixels and measure how much each layer's output changes under resampling.

Baselines

The naive reference is the unmodified backbone at matched depth: ResNet-50 for classification and a dilated ResNet-50 segmentation head for scene parsing. It tells us whether any global mixer helps.

The first strongest competitor is the global Fourier filter of Global Filter Networks for Image Classification. It has the same $O(N\log N)$-or-better cost profile and the same drop-in role, and it reports 78.6% top-1 for GFNet-XS. We re-run it in our training recipe, with an identical optimizer, schedule, augmentation, and hyperparameter search budget. We also run a zero-padded variant, which removes wrap-around. Beating it means higher top-1 at equal parameters and FLOPs, and clearly less degradation at 384 and 512 test resolution. For the zero-padded variant, there is no reported result found on Lacuna for a zero-padded GFNet.

Fast Fourier Convolution is the matched baseline in the ResNet-50 setting, with 77.8% top-1 reported. We reproduce it with the same $\alpha=0.25$ channel split so the local branches are identical.

For dense prediction, the strongest global-context competitor in the file is CCNet. It reports 45.76% mIoU on ADE20K validation and 81.9% mIoU on the Cityscapes test set, with the same two-hop global routing we use. We reimplement its module on our backbone and training schedule. We quote its published numbers only as an external reference. Beating it means equal or higher mIoU at lower memory and latency, measured on the same GPU.

The ablation that isolates the key ingredient is a constant-coefficient version of our own layer. With constant $a$ and $c$ and Neumann boundaries, the operator is diagonal in the discrete cosine basis, so it reduces to a cosine-domain global filter with a screened-Poisson shape. The gap between the constant and variable-coefficient layers measures what position dependence buys. Further ablations fix the coefficients instead of predicting them from content, and sweep $K$ and $d$.

Why not just…?

Why not just zero-pad the FFT in GFNet to kill wrap-around? It is cheap, and it probably fixes most border artifacts. That is exactly why boundary handling is not our headline. Zero padding still leaves an operator that is diagonal in frequency, identical at every position, and tied to grid size. The zero-padded GFNet baseline tests this directly. If it matches our layer on ADE20K and on resolution transfer, the proposal's claim fails.

Why not just use a cosine-transform filter, which is already non-periodic? This is the real threat. It is our constant-coefficient ablation in disguise. If variable coefficients add nothing over it, the honest conclusion is that a cosine global filter suffices. We would report that as the result.

Why not just add positional embeddings to a Fourier filter? Additive embeddings change what goes into a shift-invariant operator. They do not change the operator's reach at each location. We include GFNet with learned 2D positional embeddings in the classification runs. Our method must win on segmentation of border-adjacent and large "stuff" regions, not only on average mIoU.

Why not just use oversized kernels or criss-cross attention? ParCNetV2 needs custom implicit-GEMM kernels, and its weight count grows with resolution. CCNet costs $O(N\sqrt N)$. Our comparison against CCNet reports mIoU alongside measured memory and latency at 512×1024 Cityscapes crops. That settles whether our efficiency argument holds in practice and not just asymptotically.

Decisive Checks

The first decisive check is on ADE20K and Cityscapes. The variable-coefficient layer must beat both the zero-padded Fourier filter and its own constant-coefficient ablation, at matched cost and in the same codebase as CCNet.

The second is a resolution test. Models trained at 224 and tested at 512 must lose less accuracy than GFNet. Layer-output discrepancy under resampling should shrink as $K$ grows, whereas GFNet's interpolated filters should not show this trend.

The third checks the stated mechanism. Measured output-coefficient decay should exceed input decay by roughly $k^{-2}$. Accuracy should saturate at small $K$, around 16 to 32 modes.

If variable coefficients do not help on the first check, the idea reduces to a known cosine filter and should be dropped.