Analytic Learning with Closed Form Matrix Solutions

This direction is split into other directions: analytical weight optimization, extreme learning machine. Its last version stays readable and citable here.

Version 1 of 1281 papers365 conceptsfrozen 2026-10-04

Training a neural network is traditionally viewed as an iterative optimization problem. We initialize weights randomly, pass data through the network, calculate an error, and then use backpropagation to "nudge" the weights in a direction that reduces that error. This process, while effective, is computationally expensive, requires careful tuning of hyperparameters like learning rates, and often suffers from instabilities like vanishing gradients or catastrophic forgetting.

A growing body of research proposes a radical alternative: Analytic Learning. Instead of iteratively searching for the best weights, we can treat the training of neural layers as a system of linear equations and solve for the optimal weights in a single mathematical step.

Training as a Deterministic Linear System

The core intuition behind analytic learning is that many neural network components—especially the final classification layers or linearized hidden layers—behave like linear regressors. In a standard layer, we have an input matrix $X$ and a target matrix $Y$. We want to find a weight matrix $W$ such that $WX \approx Y$.

In gradient-based learning, we approximate $W$ over thousands of steps. In analytic learning, as explored in Analytic Learning of Convolutional Neural Network For Pattern Recognition, we solve this directly. By using the Moore-Penrose pseudoinverse ($X^\dagger$), we can calculate the optimal weights as $W = YX^\dagger$. This approach, championed by researchers like Kar-Ann Toh and Zhengguo Li in their work on KARnet, transforms training from a stochastic "guess-and-check" process into a deterministic, one-shot calculation.

The Power of the Pseudoinverse

Why use a pseudoinverse instead of a standard matrix inverse? Most data matrices are not square or invertible. The Moore-Penrose pseudoinverse provides the best possible "least-squares" fit even when the system is over-determined (more data than features) or under-determined.

This mathematical shortcut offers several immediate advantages:

  1. Speed: Ke Wang and colleagues demonstrated in PILAE that analytic solutions can be orders of magnitude faster than backpropagation, sometimes reducing training times from days to minutes.
  2. Stability: Because the solution is closed-form, it eliminates the variance caused by random weight initialization. You get the same optimal weights every time.
  3. No Hyperparameters: There is no learning rate to decay, no momentum to tune, and no early stopping required. The math simply finds the global minimum for that layer's objective.

Scaling to Deep and Broad Architectures

While solving a single layer is straightforward, deep networks present a challenge: how do we solve for weights when the "targets" for internal layers are unknown?

One popular strategy is the Broad Learning System (BLS), developed by C.L.Philip Chen. Instead of going deep, BLS goes "wide," mapping inputs into a high-dimensional feature space and solving for the output weights analytically. As shown in Rich Feature Combination for Cost-Based Broad Learning System, this allows for rapid incremental learning—adding new features or data points only requires updating the existing matrix solution rather than retraining the whole model.

For deep architectures, researchers use "layer-wise" analytic training. Rui Zhang and Hongyuan Zhang introduced the Non-Gradient Manifold Neural Network (NGMNN), which uses invertible activation functions to propagate targets backward, allowing each layer to be solved as a local regression problem. Similarly, DensePILAE by Yanjun Li uses feature reuse and pseudoinverse learning to train deep stacked autoencoders without a single gradient step.

Absolute Memory and Single-Round Aggregation

Analytic learning is particularly transformative for two of AI's biggest hurdles: Continual Learning and Federated Learning.

In continual learning, gradient descent often causes "catastrophic forgetting" as new updates overwrite old knowledge. However, Songning Lai and the creators of CONCIL show that by using recursive matrix operations, a model can achieve "absolute knowledge memory." Instead of overwriting, the model mathematically integrates new data into its existing covariance matrices, ensuring it never forgets previous concepts.

In Federated Learning, the bottleneck is the constant communication of gradients between clients and a central server. Huiping Zhuang and Run He proposed Analytic Federated Learning (AFL). Since the weights are derived from data statistics (like the autocorrelation matrix), clients can simply send their local statistics to the server. The server then sums these matrices—a process called the "Absolute Aggregation Law"—to compute the exact same model that would have resulted from centralized training, all in a single communication round.

The Hybrid Future

We are seeing a shift toward hybrid regimes. Closed-Form Last Layer Optimization suggests that even if we use gradients for the "backbone" of a network, we should always solve the final linear layer analytically at each step to ensure the fastest possible convergence.

By moving away from pure iterative "guessing" and toward exact mathematical solutions, analytic learning provides a path toward AI that is not only faster and more efficient but also more stable and interpretable.

Papers in this version

Showing 60 of 281 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_analytic_learning_with_closed_form_matri_f4bec98b,
  title = {Analytic Learning with Closed Form Matrix Solutions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/analytic-learning-with-closed-form-matrix-solutions/rdr_f4bec98bfba84bfbb514a2d16c45a81e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_analytic_learning_with_closed_form_matri_f4bec98b_v1,
  title = {Analytic Learning with Closed Form Matrix Solutions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/analytic-learning-with-closed-form-matrix-solutions/rdr_f4bec98bfba84bfbb514a2d16c45a81e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}