Variational Functional Neural Networks (vfNN)

Chinmay Katke, C. Nadir KaplanarXiv 2026 · arXiv

In the rapidly evolving field of scientific machine learning, Physics-Informed Neural Networks (PINNs) have become the go-to tool for solving differential equations. However, PINNs come with baggage: they often require computing high-order derivatives and involve a "balancing act" of tuning ad hoc weight factors for different terms in the loss function.

A new paper by Chinmay Katke and C. Nadir Kaplan introduces Variational Functional Neural Networks (vfNN), a framework that sidesteps these issues by returning to the foundational principles of physics: the variational formulation.

TL;DR

vfNN finds physical states by directly optimizing integral functionals (like energy or action) rather than solving differential equations. By using a custom quasi-Newton optimizer with simulated annealing, it achieves faster convergence than PINNs and can identify both stable and unstable physical solutions that traditional methods miss.

What Problem Does This Paper Address?

Most physical systems are governed by a principle of extremization—nature likes to minimize (or sometimes maximize) quantities like energy, action, or entropy. Traditionally, we use the calculus of variations to derive differential equations (Euler-Lagrange Equations, or ELE) from these functionals and then solve the equations.

PINNs follow this "derived" path: they minimize the residual of the differential equations. This approach has two major drawbacks:

  1. Computational Cost: Solving a second-order ODE requires computing third or fourth-order derivatives during backpropagation, which is computationally expensive.
  2. Hyperparameter Hell: PINN loss functions are often a sum of several terms (equation residuals, boundary conditions) multiplied by weight factors ($\mu_i$) that must be tuned manually or via complex algorithms.

The authors ask: Why solve the derivative equations when we can optimize the original functional directly?

Methods

The vfNN architecture uses a standard Multilayer Perceptron (MLP) to represent the unknown function $u(x)$. The "magic" happens in the loss function and the optimizer.

1. The Variational Loss

Instead of an $L^2$ norm of an ODE residual, the loss function is a discrete approximation (a Riemann sum) of the physical functional itself. vfNN Architecture Figure 1: The vfNN architecture directly maps inputs to a functional $\mathcal{J}$ through a neural network.

vfNN Architecture

2. Hard-Coded Boundary Conditions

Unlike PINNs, which often treat boundary conditions as penalty terms in the loss, vfNN implements Dirichlet conditions by transforming the network output: $u(x) = \tilde{l}(x)u_N(x) + g(x)$. This ensures the boundary conditions are satisfied exactly by construction.

3. The SR1 Optimizer with Simulated Annealing

Directly optimizing functionals is tricky because the "loss landscape" isn't always a nice, convex bowl. Physical solutions can sit at minima, maxima, or saddle points. Standard optimizers like ADAM or BFGS (which assumes a positive-definite Hessian) often fail here.

The authors developed a custom optimizer combining:

  • Symmetric Rank 1 (SR1) Updates: Unlike BFGS, SR1 can handle indefinite Hessians, allowing it to find saddle points.
  • Stochastic Line Search: A Metropolis-criterion-based search that helps the algorithm navigate rough, non-convex landscapes.
  • Simulated Annealing: A "quenching" process that gradually narrows the search to converge on the critical point.

Datasets

The method was evaluated on several classic physical benchmarks governed by Ordinary Differential Equations (ODEs):

  • Linear ODEs: Various forms of $u'' = f(u, x)$.
  • Minimal Surface of Revolution: Finding the shape of a soap film between two rings (a classic non-convex problem).
  • Sine-Gordon Equation: Models pendulums and dislocations in crystals.
  • Poisson-Boltzmann Equation: Describes electrostatic potential in electrolytes.
  • Colloidal Membrane Packing: A complex system of coupled nonlinear ODEs describing chiral hard rods.

Key Insights and Ideas

  • Lower Order is Better: Because vfNN works with the functional, it only needs first-order derivatives where a PINN would need second-order. This significantly reduces the computational burden of automatic differentiation.
  • Physical Weighting: The "weights" in the vfNN loss are not arbitrary; they are the actual physical constants of the system. This removes the need for heuristic weight tuning.
  • Stability as a Feature: The non-convexity of the loss landscape in vfNN directly corresponds to the physical stability of the system. The presence of "conjugate points" (where solutions become unstable) is a physical reality that vfNN can navigate.
  • Finding the Unstable: In systems like the Sine-Gordon equation, vfNN successfully found unstable solutions (saddle points) that the ADAM optimizer simply could not locate.

Key Results

The authors compared vfNN against PINNs using both ADAM and their SR1 optimizer.

  • Speed: For linear ODEs, vfNN with SR1 converged in significantly fewer iterations than PINN counterparts. For example, in one benchmark, vfNN took 28 iterations where PINN took 120.
  • Accuracy: vfNN results showed excellent agreement with Finite Element Method (FEM) solutions across all nonlinear benchmarks. Nonlinear ODE Results Figure 3: vfNN successfully identifies multiple solutions (stable and unstable) for the minimal surface and Sine-Gordon problems.
FIG. 3. Nonlinear ODEs using vfNN. Solutions to the functionals whose ELE are (a) the minimal surface of revo- lution (Eqs. 24, 25), (b) steady-state Sine-Gordon equation (Eq. 26), and (c) the Poisson-Boltzmann equation (Eqs. 27– 29). The dashed red lines indicate the solutions using vfNN, and all other colors correspond to different nonunique solu- tions (a) from the exact solution (Eq. 25) and (b), (c) found by using FEM. For all panels, a deep neural network with three layers and 20 neurons at each layer is used.
  • Robustness: In the colloidal membrane problem (Figure 4), vfNN accurately captured the coupling between tilt angle $\theta(x)$ and order parameter $\psi(x)$, matching FEM data nearly perfectly.

Limitations and Future Work

The current implementation focuses on Ordinary Differential Equations (ODEs). While the principles are general, extending vfNN to Partial Differential Equations (PDEs) requires treating space and time variables on equal footing within the functional, which is the authors' stated next goal.

Additionally, while the SR1 optimizer is powerful, the algorithm currently requires some manual tuning of the "quenching rate" in the simulated annealing process. Automating the detection of conjugate points to assess solution stability during training remains an open challenge.

Summary updated

Abstract

We present a variational functional neural network (vfNN) algorithm that optimizes physically motivated continuum functionals in integral form (e.g., Lagrangian, Hamiltonian, or free energy) to find the states of a system. That way, vfNN bypasses solving the equations of motion (a.k.a. the Euler-Lagrange equations), which are in the form of differential equations. To this end, our approach is fundamentally different from conventional physics-informed neural networks (PINN) that solve differential equations by minimizing a phenomenological loss function involving higher order derivatives and ad hoc weight factors. In contrast, vfNN merely evaluates lower order derivatives of the generating functional and replaces the weight factors with rigorous physical scales. This also fundamentally changes the loss landscape, and the desired solutions may become unstable. For this emergent nonconvex optimization problem, we develop a quasi-Newton optimization algorithm with simulated annealing that converges to the solutions of the corresponding Euler-Lagrange equations of the functional regardless of their stability properties. Our simulations using vfNN give accurate results for benchmark systems governed by linear or nonlinear ordinary differential equations.
BibTeX
@misc{katke2026variational,
  author = {Chinmay Katke and C. Nadir Kaplan},
  title = {{Variational Functional Neural Networks (vfNN)}},
  year = {2026},
  eprint = {2611.03542},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03542},
}
Figure