In standard machine learning, we have a habit of "flattening" the world. Whether it is a 2D image, a 3D medical scan, or a 4D convolutional kernel, we typically collapse these multi-dimensional structures into one-dimensional vectors or two-dimensional matrices before processing them.
While this makes the linear algebra convenient, it comes at a steep cost: it destroys the structural integrity of the data. When you flatten a tensor, pixels that were neighbors in 3D space are suddenly scattered across a long vector. The model is then forced to use its limited parameter budget to "relearn" spatial and temporal relationships that were already explicitly present in the original format.
A growing body of research suggests a different path: treating data and parameters as tensors throughout the entire pipeline. By using multilinear algebra instead of flat linear algebra, we can preserve the topological relationships of the data, leading to models that are more efficient, robust, and expressive.
The Cost of Flattening
The fundamental problem with "matricization" (turning a tensor into a matrix) or "vectorization" is the loss of spatial locality. As noted by Chongjie Si and Xuehui Wang in Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning, reshaping high-dimensional weights into 2D matrices for adaptation compromises the effectiveness of layers that depend on the relative positioning of elements, such as pixels or channels.
This isn't just a problem for neural network weights. In climate forecasting or traffic analysis, Tajamul Ashraf argues in FATE: Focal-modulated Attention Encoder for Temperature Prediction that flattening multidimensional data destroys the hierarchical relationships between time, space, and features. When the structure is lost, the model struggles to capture how variables interact across different scales.
Multilinear Operators as a Solution
To avoid the "flattening tax," researchers are replacing standard layers with tensor-native operators.
A prime example is NdLinear Is All You Need for Representation Learning by Alex Daniel Reneau and colleagues. Instead of a standard linear layer ($y = Wx$), NdLinear applies transformations dimension-wise. This allows the model to process multi-dimensional tensors natively, achieving parameter reductions of up to 95% because it doesn't need a massive, redundant weight matrix to represent every possible cross-dimensional interaction.
Similarly, the Higher-Order Transformer (HOT), introduced by Soroush Omranpour, Guillaume Rabusseau, and Reihaneh Rabbany, factorizes the attention mechanism. Instead of attending over a flattened sequence, HOT uses mode-wise matrices (often via Kronecker products or sums) to capture dependencies. This preserves the intrinsic structure of data like 3D medical scans while significantly lowering computational costs Higher Order Transformers: Efficient Attention Mechanism for Tensor Structured Data.
Turning the Curse into a Blessing
In classical statistics, high dimensionality is often seen as a "curse" because the number of parameters grows exponentially with the number of dimensions. However, Ilia Kisil and Giuseppe G. Calvi argue that by using tensor decompositions, we can turn this into a "blessing" Tensor Valued Common and Individual Feature Extraction: Multi-dimensional Perspective.
The key mechanism is Tensor Decomposition (such as Tucker or CP decomposition). These methods represent a large, complex tensor as a product of much smaller "core" tensors and factor matrices.
- Efficiency: Xian Wei and others showed in Learning Robust and Lightweight Model through Separable Structured Transformations that decomposing weight matrices into Kronecker products can achieve 200x compression while increasing adversarial robustness.
- Classification: Tensor LDA-MD leverages Tucker low-rank structures to classify high-dimensional data even when many entries are missing, achieving theoretically optimal misclassification rates that vector-based methods cannot reach.
Structural Integrity in Fine-Tuning
This philosophy is now being applied to the fine-tuning of Large Language Models and Vision Transformers. Standard Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA often reshape 4D tensors into 2D matrices to apply low-rank updates.
The FLoRA framework Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning and the LieRA framework Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformations by Chongjie Si argue that this reshaping is a mistake. By using Tucker decomposition or Lie group transformations, these methods ensure that the fine-tuning process respects the original geometry of the parameter space. This allows the model to learn complex spatial correlations more effectively than simple matrix-based adaptation.
Summary of Intuition
The shift toward tensor-native learning is based on a simple realization: geometry is information. 1. Flattening is destructive: It removes the "spatial context" that tells the model which data points are related. 2. Multilinear algebra is structural: Operators like the Tucker or Kruskal operators Multilinear operators for higher-order decompositions. allow us to manipulate data along its natural axes (e.g., Space $\times$ Time $\times$ Channel). 3. Decomposition is a regularizer: By forcing the model to represent information as a low-rank tensor, we prevent overfitting and reduce parameter counts without losing the ability to model complex, higher-order interactions.