Learned Domain Splits for Physics-Constrained Mixture-of-Experts

Neural networks that solve partial differential equations (PDEs, the equations governing fluids, heat, and transport) can be forced to obey the physics exactly by attaching a small numerical solver to their output. That solver is expensive. A recent fix splits the simulated domain into pieces and gives each piece its own "expert", a local solver that enforces the equations only inside its patch. The split, however, is a fixed uniform grid. A quiet corner of the fluid gets as much solver effort as a turbulent vortex or a singularity. This proposal learns where the patch boundaries go. It does so in a way that survives the two failures that usually come with learned splits. The first is expert collapse: the model that assigns points to experts starves most of them and sends all work to a few. The second is the discretization gap: the soft, differentiable assignment used in training stops being valid once it is snapped to a hard partition at test time.

The core idea fits in two sentences. Experts should be balanced by how much numerical work they do, measured as local PDE difficulty, and not by how many grid points they receive. The soft assignment should be pushed gradually to a hard one during training with a concave penalty, so the hard constraints the experts enforce hold for the partition actually used at inference. If this works, physics-constrained neural solvers get compute that adapts to where the physics is hard, and a learned split stays tied to the solver it feeds.

Core Claim

Scaling physics-informed hard constraints with mixture-of-experts (PI-HC-MoE) is the method this proposal builds on, and its result is the gap this proposal targets. In PI-HC-MoE, a Fourier Neural Operator backbone produces basis functions. A static grid splits the domain into $K$ subdomains. Each expert solves a local nonlinear least-squares problem so that the PDE holds on its patch, and gradients pass through each solver by implicit differentiation. It reports about 3.6% relative error on 1D Diffusion-Sorption, against about 7.5% for a single global hard constraint. The authors name adaptive partitioning as the open problem.

The obvious next step is to drop in a learned spatial router of the kind used by Spatial Mixture-of-Experts. That paper routes each grid cell to an expert and stabilizes routing with a self-supervised routing loss and gradient damping, but it attaches no numerical solver. We claim this naive combination fails in a specific, predictable way, and that two changes fix it.

The first failure is collapse. Stiff regions, such as the $u=0$ singularity in Diffusion-Sorption, produce poorly conditioned local solves and noisy gradients. We hypothesize that an unconstrained router will move probability away from those regions' dedicated experts. This is a hypothesis, not an established result: the file's Gap 2 asserts it, but no paper in the corpus measures it.

The second failure is the gap. With soft routing, each point's prediction is a weighted blend of several experts' local solutions. Each local solution satisfies the PDE, but for a nonlinear PDE a blend generally does not. Hardening the assignment by taking the highest-probability expert changes which points each local solver sees. The trained constraint guarantees therefore do not transfer to the hard partition.

Research Program

Let $p_{ie}$ be the router's probability of sending collocation point $i$ to expert $e$. During training, expert $e$'s local least-squares solve weights point $i$ by $p_{ie}$, and the prediction at $i$ is $\sum_e p_{ie} u_e(x_i)$. Let $c_i$ be a per-point difficulty proxy. Candidates are the magnitude of the backbone's unconstrained PDE residual, or the local Jacobian norm of the solver's residual. The proxy is computed without gradients so the router cannot game it.

Expert load is then defined as $L_e = \sum_i p_{ie} c_i$, which counts work rather than points. Load is balanced with the population-level scheme of $\phi$-Balancing for Mixture-of-Experts Training. That scheme turns a convex potential on expected load into "congestion prices" $\mu_e$, maintained by mirror descent on an exponential moving average of historical load. Its authors use it for counts of language tokens with identical experts. We reuse their machinery but change what is measured.

The training objective is

$$\mathcal{L} = \mathcal{L}_{\text{task}} + \sum_e \mu_e L_e + \lambda_t \sum_i \Big(1 - \sum_e p_{ie}^2\Big),$$

where $\mathcal{L}_{\text{task}}$ is the data and solution loss and $\lambda_t$ is a penalty weight increased over training. The last term is the simplex form of the concave penalty from Relax and penalize. That paper uses $\sum_j \theta_j(1-\theta_j)$ for binary hyperparameters and proves that minimizers coincide with discrete ones once the penalty is large enough. Our term is nonnegative and zero exactly when every point's assignment is one-hot. Early training therefore explores soft splits, and late training lands on a hard partition that the local solvers have already been fitted to. When the penalty reaches its final weight, assignments are effectively discrete. The inference partition is then the training partition and no rounding step remains.

The schedule for $\lambda_t$ follows a plateau-triggered rule. It is borrowed from PRDP, which increases solver iterations only when the loss stalls. That paper shows coarse early fidelity loses no final accuracy on 2D Navier-Stokes while cutting solver iterations by 81%. This is the main constraint taken from prior work: annealing schedules are a known source of instability, and Relax and penalize itself warns that the concave term can trap optimization in poor discrete minima.

Beyond the method, the program has a diagnostic arm. We will log router mass near known stiff features, the number of dead experts, the load Gini coefficient, and expert parameter similarity across training. These logs test whether physics-driven collapse actually happens. Similarity checks the homogenization that over-strict balancing causes in Advancing Expert Specialization for Better MoE. We will also log the "hardening gap", defined as the error and PDE residual of the soft model minus those of its argmax-hardened version, at every checkpoint.

Baselines

The do-nothing reference is the global hard-constraint model from Scaling physics-informed hard constraints with mixture-of-experts, with one solver over the whole domain. It reports about 7.5% relative error on 1D Diffusion-Sorption. Any decomposition must beat this or it has no reason to exist.

The strongest existing method is PI-HC-MoE itself with its static uniform grid, reported at about 3.6% relative error on Diffusion-Sorption. For 2D Navier-Stokes at Reynolds number $10^4$, the file gives only a 2.5x to 12.8x speedup over global constraints and no absolute error, so we report "no reported result found on Lacuna for PI-HC-MoE relative error on 2D Navier-Stokes". We will rerun it ourselves. Fairness here means the same backbone and the same number of experts $K$. It also means the same least-squares solver and iteration cap, the same collocation points, the same wall-clock budget, and the same number of tuning trials for every method. Beating it means lower relative error at equal $K$ and equal solver work, or equal error with measurably less total solver work. The margin must exceed seed variance across five seeds.

We will add a stronger version of the static baseline: a uniform grid with more experts, chosen so that its total solver cost matches ours. This tests whether adaptivity, rather than raw capacity, explains any gain.

The non-learned adaptive competitor is a residual-driven partition in the style of classical adaptive mesh refinement. Every few epochs, a quadtree or k-means split over the backbone's residual field is recomputed so that each patch holds equal summed difficulty $c_i$. It gets the same difficulty signal and the same recomputation budget.

Two learned-router baselines isolate our ingredients. One is a learned router with a Switch-style batch balancing loss and argmax at test time. The other is a learned router with count-based $\phi$-balancing and argmax.

The ablations swap one piece each. Difficulty-weighted load is replaced by count-weighted load. The annealed penalty is replaced by plain argmax rounding, and separately by straight-through hard routing from the start. The difficulty proxy is replaced by a random field, to check that the proxy, and not the extra regularizer, drives the effect.

Why not just…?

Why not just use a finer static grid? This is a real threat on smooth problems. If the hard regions are spread out, a uniform grid with more experts at matched cost may match any learned split. The matched-cost static baseline settles this. We expect our advantage to grow as the hard regions become more localized, so we report results separately for the localized singularity in Diffusion-Sorption and for diffuse turbulence. If the advantage does not grow with localization, adaptivity is not paying for itself.

Why not just partition by the residual, as in adaptive mesh refinement, with no learned router? This is the most serious alternative, because it uses the same difficulty signal without any of the learning machinery. It has two weaknesses. Its partition jumps discontinuously at each recomputation, so the local solvers are refitted to new patches mid-training. It also cannot trade difficulty against how well the backbone's basis functions fit a patch, which only the task gradient sees. Both are claims to test, not facts. If the residual-driven partition matches our method within seed variance, we will say so, and the contribution narrows to the collapse and hardening-gap diagnostics plus the finding that a simple heuristic suffices.

Why not just add the standard auxiliary balancing loss and take argmax at test time? Batch balancing forces equal point counts, which is the wrong target when difficulty is uneven. It is also known to make experts nearly identical, as shown by Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer and Advancing Expert Specialization for Better MoE. Argmax rounding of relaxed solutions is the failure documented by Relax and penalize, with top-$k$ rounding at 24.00 RMSE against 20.87 for penalization on BlogFeedback distillation. That evidence comes from feature selection, not PDEs. The hardening-gap measurement on our baselines is what would show whether it carries over.

Why not just use count-based $\phi$-balancing off the shelf? It already avoids batch noise and lifted Moonlight-16B-A3B on Math500 from 64.8% to 66.2%. But it assumes every token costs the same, and that assumption is exactly what fails here. The count-versus-difficulty ablation is the direct test. If the two tie, the difficulty weighting is unnecessary and we drop it.

Proposed Experiments

All experiments use PDEBench (PDEBench: An Extensive Benchmark for Scientific Machine Learning). The two primary tasks are 1D Diffusion-Sorption and 2D turbulent Navier-Stokes at $Re=10^4$, the settings where PI-HC-MoE reports its results. Metrics are relative error together with PDEBench's conservation (cRMSE), boundary (bRMSE), and Fourier-spectrum (fRMSE) errors. We also report the PDE residual of the final hard partition, total solver iterations, wall-clock time, dead-expert count, and the load Gini coefficient. Partition maps are compared against residual maps to check that capacity follows difficulty.

Three checks decide the outcome. First, under unconstrained routing, does router mass drain from experts covering the $u=0$ singularity within the first epochs? If it does not, the physics-specific collapse hypothesis is wrong, and the balancing component must be justified on efficiency grounds alone. Second, does the annealed penalty drive the hardening gap in PDE residual to near zero while argmax-rounded baselines keep a gap? This is the discretization claim, and it is falsifiable checkpoint by checkpoint. Third, at matched $K$ and matched solver work, does the full method beat both the about 3.6% static PI-HC-MoE figure we reproduce and the residual-driven heuristic partition? If it beats the former but not the latter, the honest conclusion is that adaptivity matters but learning the split does not.