Kronecker Product Factorization for Parameter Efficient Networks

This direction is split into other directions: low-rank weight factorization (low-rank matrix factorization). Its last version stays readable and citable here.

Version 1 of 1113 papers130 conceptsfrozen 2026-10-04

As neural networks scale to billions of parameters, the weight matrices in fully-connected layers and embeddings have become a primary bottleneck for deployment on edge devices. Traditional compression often relies on low-rank approximation (like SVD), which reduces the number of parameters by assuming the weight matrix is "thin" in its latent space. However, a more powerful family of techniques—centered on the Kronecker product—is emerging to compress models by exploiting structural redundancy without sacrificing the "rank" or expressive capacity of the network.

The Kronecker Trick: Structure over Sparsity

The core intuition behind Kronecker-based compression is to represent a massive weight matrix $W$ not as a collection of independent numbers, but as the interaction between two or more much smaller matrices. Mathematically, if $A$ is an $m \times n$ matrix and $B$ is a $p \times q$ matrix, their Kronecker product $W = A \otimes B$ results in a large $(mp) \times (nq)$ matrix where every element of $A$ is multiplied by the entire matrix $B$.

Shuchang Zhou and colleagues in Compression of Fully-Connected Layer in Neural Network by Kronecker Product demonstrated that this allows for extreme parameter reduction. For example, a $4096 \times 4096$ matrix (16 million parameters) can be represented by two $64 \times 64$ matrices (only 8,192 parameters). This isn't just about making the model smaller; it’s about forcing the model to learn a structured transformation that repeats and scales local patterns across the global weight space.

Why Kronecker Beats Low-Rank Factorization

In standard low-rank factorization ($W = UV^T$), the resulting matrix $W$ is limited by the rank of the inner dimension. If you compress a matrix significantly, you inevitably lose the ability to model complex, high-frequency interactions.

Urmish Thakker and the team at Arm ML Research argue in Pushing the limits of RNN Compression that Kronecker products are fundamentally different. Because the rank of a Kronecker product is the product of the ranks of its factors ($\text{rank}(A \otimes B) = \text{rank}(A) \cdot \text{rank}(B)$), a highly compressed Kronecker-factored matrix can still be full-rank. This allows models like RNNs to maintain high expressive power and training stability even at 38x compression ratios, where SVD-based models typically collapse.

Basis Decomposition for Multi-Relational Data

The Kronecker concept extends naturally to scenarios where a model must handle many different types of relationships, such as in Knowledge Graphs or Multi-Task Learning. Instead of learning a completely separate weight matrix for every relation or task—which leads to "parameter explosion"—researchers use basis decomposition.

In this setup, the weight matrix for a specific relation is represented as a linear combination of a small set of shared "basis" matrices. Dan Busbridge applied this to Relational Graph Attention Networks, showing that it forces the model to learn commonalities across different relationship types. Similarly, Junlong Du developed the VMT-Adapter, which uses a Kronecker-based strategy to share core projection layers across multiple dense scene understanding tasks, achieving better performance than full fine-tuning with only 1% of the parameters.

Stacking and Reshaping for Expressivity

While a single Kronecker product is efficient, it can be rigid. Recent work has focused on making these factors more flexible:

  1. Sums of Kronecker Products: Aliakbar Panahi introduced Shapeshifter, which reshapes weight matrices before applying factorization. This is mathematically equivalent to a sum of Kronecker products, which the authors prove increases the model's expressiveness beyond simple low-rank methods.
  2. Hierarchical Factorization: To achieve even higher efficiency in bilinear pooling, Tan Yu and researchers at Baidu proposed Two-level Kronecker Product Factorization (TKPF). By nesting Kronecker products and integrating identity matrices, they achieved a 53x speedup in projection layers, effectively "thinning" the computation at multiple levels of abstraction.

Practical Impact: From LLMs to IoT

The shift toward structured factorization is enabling a new generation of "edge-ready" models. In the realm of Large Language Models, Wonkyo Choe demonstrated in RWKV-edge that combining low-rank decomposition with structural constraints allows LLM-class performance to run on low-power hardware like the Orange Pi.

By moving away from the "dense-is-better" paradigm and embracing the mathematical elegance of the Kronecker product, researchers are proving that the "knowledge" within a neural network is often highly structured—and that we can capture that structure with a fraction of the memory.

Papers in this version

Showing 60 of 113 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_kronecker_product_factorization_for_para_c26b2f7e,
  title = {Kronecker Product Factorization for Parameter Efficient Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/kronecker-product-factorization-for-parameter-efficient-networks/rdr_c26b2f7ed978483889eaef8b1429dda1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_kronecker_product_factorization_for_para_c26b2f7e_v1,
  title = {Kronecker Product Factorization for Parameter Efficient Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/kronecker-product-factorization-for-parameter-efficient-networks/rdr_c26b2f7ed978483889eaef8b1429dda1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}