Standard deep learning architectures treat neural network weights as static matrices. Once training is complete, a layer applies the exact same linear transformation regardless of the context, the depth of the computation, or the progression of time. When faced with dynamic tasks—such as modeling physical systems, tracking evolving data streams, or executing deep recursive reasoning—the traditional workaround is to append a temporal or structural coordinate directly to the input vector.
However, forcing a static set of weights to process time-varying inputs requires the network to learn highly complex, non-linear coordinate mappings. A powerful alternative paradigm has emerged: treating the weights themselves as continuous trajectories. By parameterizing a network's weights as continuous functions of time, depth, or phase ($W = W(t)$), the network becomes a nonautonomous dynamical system. This allows its internal transformation logic to morph smoothly, offering a more natural and expressive foundation for modeling continuous phenomena.
Moving Beyond Static Parameters
The mathematical roots of this idea trace back to optimal control theory. Instead of viewing optimization as a search for a single static point in parameter space, it can be framed as finding an optimal trajectory through that space. Early work by Farotimi et al. demonstrated that evolving a network's weights during its state transitions could reliably guide the system toward desired attractors, transforming weight synthesis into a rigorous control problem.
When neural networks are extended to continuous depths—as seen in Neural Ordinary Differential Equations (Neural ODEs)—this perspective becomes essential. Taghvaei et al. formalized this by modeling network layers as continuous-time dynamical systems where weights are represented as time-varying matrices across the network's depth.
Input Concatenation (Static Weights):
x(t) ---> [ Static Weights W ] ---> x(t+1)
^ (Time appended to input)
Continuous Weight Trajectory:
x ---> [ Dynamic Weights W(t) ] ---> x(t+1)
^ (Weights morph continuously with t)
This shift from static parameters to continuous trajectories fundamentally alters how a network generalizes. As Marion demonstrated, the generalization bounds of continuous-depth models are governed not by their total depth or parameter count, but by the smoothness—specifically the Lipschitz constant—of their weight trajectories. Smooth trajectories ensure that the network's transformation logic does not oscillate wildly, leading to more stable and predictable behavior.
Mechanisms for Continuous Weight Generation
To implement a continuous weight trajectory $W(t)$ without storing an infinite number of matrices, models must utilize compact, differentiable parameterizations. Three primary mechanisms have emerged to achieve this:
1. Time-Dependent Weight Modulation
Instead of feeding a temporal coordinate into the network as a feature, the coordinate is used to dynamically scale or shift the layer weights. Park et al. showed that this architectural conditioning is fundamentally superior to input concatenation for capturing continuous physical transformations. It allows the network to preserve its spatial representation capacity while smoothly adapting its transformation logic across time.
2. Hypernetworks
A secondary network can be trained to map a scalar coordinate (such as time) directly to the high-dimensional weight space of a primary network. For example, Rotman et al. utilized a hypernetwork to generate the spatial operator weights of a Fourier Neural Operator (FNO) as a continuous function of time. This enables the model to interpolate the state of a dynamical system at any arbitrary intermediate time step without being constrained to discrete simulation intervals.
3. Continuous Low-Rank Adaptation
To scale continuous trajectories to massive models, researchers combine base static matrices with time-varying low-rank updates. Berman et al. introduced CoLoRA, which modifies a base weight matrix using a low-rank update scaled by a continuous, time-dependent parameter. Similarly, Lu et al. developed a time-varying low-rank adapter (Terra) that inserts a differentiable, time-dependent middle matrix between low-rank layers. This design constructs a smooth parameter manifold that allows models to transition seamlessly between distinct domains.
Solving Evolution Equations in Parameter Space
One of the most impactful applications of continuous weight trajectories is in solving time-dependent Partial Differential Equations (PDEs). Traditional physics-informed neural networks attempt to learn a global mapping from space-time coordinates to solution values, which often requires expensive retraining if the initial boundary conditions change.
By contrast, an evolutionary parameter approach uses the neural network to represent the spatial state of the system at a single instant, and then evolves the network's parameters forward in time. As Feng et al. and Mou et al. illustrated, this maps the evolution of a physical system from an infinite-dimensional function space onto a trajectory within the finite-dimensional parameter space of the network.
Physical Space Evolution:
State(t=0) ------------------------> State(t=T) (Infinite-dimensional field)
Parameter Space Trajectory:
Weights(t=0) ---> Weights(t) ---> Weights(T) (Finite-dimensional ODE)
To determine how these parameters should move, the physical dynamics of the governing PDE are projected onto the tangent space of the neural network's parameter manifold. Berman et al. developed a variational approach that projects these time-dependent equations step-by-step, updating a sparse subset of parameters like a classical numerical solver.
By applying the chain rule to the network's output with respect to its weights, the underlying physical laws are converted into a system of ordinary differential equations that govern the parameter updates. Gaby et al. and Kao et al. demonstrated that solving this derived ODE system allows the network to intrinsically track the temporal evolution of complex physics, enabling long-horizon extrapolation without global space-time optimization.
Compact Representations via Initial Value Problems
Representing an arbitrary weight trajectory across deep layers can quickly lead to a parameter explosion. To keep models compact and scalable, researchers leverage physics-inspired constraints to generate the entire trajectory from a minimal set of parameters.
An elegant solution is the shooting formulation of deep learning. In A shooting formulation of deep learning, the continuous trajectory of weights across the depth of a network is parameterized as the solution to an initial value problem. By defining a small set of interacting particles governed by Hamiltonian dynamics, the entire evolution of the weights across all layers is uniquely determined by their initial positions and momenta. Optimization is performed solely on these starting conditions, allowing the network to capture complex, layer-dependent transformations while maintaining a highly compact parameter footprint.
For systems that exhibit periodic or cyclic behavior, trajectories can be parameterized using basis interpolation. Sharma et al. designed an architecture where network weights are defined as a continuous manifold that varies according to a cyclic phase parameter. During the forward pass, the weights are computed dynamically by interpolating between a fixed set of learned basis weights using cubic splines. Emami et al. extended this phase-conditioned approach to multi-agent reinforcement learning, enabling policies to smoothly morph across repeating cycles without a linear increase in parameters.
Applications in Symmetry Breaking and Adaptation
Beyond physics simulation, continuous weight trajectories resolve fundamental structural limitations in standard deep architectures:
- Symmetry Breaking in Looped Transformers: Weight-tied recursive models, such as Looped Transformers, apply the exact same weight matrix across multiple computational steps. This structural symmetry limits their approximation capacity. Xu showed that introducing timestep-conditioned weight modulation breaks this symmetry, allowing weight-tied layers to adapt their behavior at different stages of computation and significantly boosting reasoning performance.
- Tracking Non-Stationary Environments: In streaming data scenarios, decision boundaries often rotate or expand over time (concept drift). Huang et al. modeled network parameters as states within a continuous-time stochastic dynamical system, allowing the model to track shifting distributions in real time without explicit test-time fine-tuning. Similarly, Kumagai used Gaussian Processes to model classifier parameters as continuous functions of time, enabling the model to extrapolate future decision boundaries without immediate access to new labels.
- Temporal Relaxation in Clinical Time-Series: Standard recurrent models assume that the relationship between clinical features and patient outcomes remains constant. Oh et al. introduced temporal parameter relaxation, allowing model weights to evolve over the duration of a sequence. This flexibility allows the network to adapt to changing underlying clinical baselines as a medical condition progresses.
By shifting the view of neural network parameters from static constants to continuous trajectories, this paradigm bridges the gap between deep learning, optimal control, and physical dynamics. Whether through hypernetworks, low-rank modulations, or initial value formulations, parameterizing weights as continuous functions allows neural networks to inherently mirror the continuous, evolving nature of the real world.
Key Foundational Work
Neural Network Weight Matrix Synthesis Using Optimal Control Techniques (1989) — contributed 1 concept elements to this direction.
Discovering Long Range Properties of Social Networks with Multi-Valued Time-Inhomogeneous Models (2010) — contributed 1 concept elements to this direction.