In traditional machine learning, the first step in processing an image or a video is often the most destructive: vectorization. To feed a $100 \times 100$ pixel image into a standard neural network, we flatten it into a single $10,000$-dimensional vector.
As noted by Kien Do in Learning Deep Matrix Representations, this process effectively "shreds" the data. It treats a pixel and its immediate neighbor the same way it treats two pixels on opposite corners of the frame. By discarding the inherent spatial and temporal relationships, we force the model to relearn the concept of "proximity" from scratch, often requiring massive amounts of data to do so.
A growing body of research suggests a more elegant alternative: treating multidimensional data as tensors (multi-way arrays) throughout the entire learning pipeline.
The Geometry of Bilinear Projections
The core intuition behind this shift is that the structure of the data should dictate the structure of the model. Instead of a standard linear transformation ($y = Wx$), researchers like Guanglei Qi and colleagues in Matrix Variate RBM and Its Applications propose using bilinear transformations.
In a matrix-variate model, an input image $X$ is transformed using two smaller weight matrices, $U$ and $V$, such that the output is $Y = UXV^T$.
- $U$ operates on the rows (capturing vertical correlations).
- $V$ operates on the columns (capturing horizontal correlations).
By processing the dimensions separately but simultaneously, the model preserves the grid-like topology of the image. This approach is extended to higher dimensions—such as 3D medical scans or video sequences—using tensor products. As Ce Zhu explains in Tensor Regression, this allows the model to capture "multi-directional correlations" that are invisible to vector-based methods.
Solving the Curse of Dimensionality
Beyond preserving structure, tensor-based methods offer a dramatic reduction in parameters. Consider a hidden layer that maps a $100 \times 100$ image to another $100 \times 100$ representation:
- Vectorized approach: Requires a weight matrix of $10,000 \times 10,000$, or 100 million parameters.
- Matrix-variate approach: Requires two matrices of $100 \times 100$, totaling only 20,000 parameters.
This efficiency is a recurring theme. Na Qi and Xiaoyan Sun demonstrate in TenSR: Multi-dimensional Tensor Sparse Representation that using "separable dictionaries" allows for massive reductions in memory and computation. Because the models have fewer degrees of freedom, they are less prone to overfitting, making them exceptionally strong in "small sample" regimes where training data is scarce.
Robustness to Noise and Occlusion
Because tensor representations respect the global geometry of the data, they are inherently more robust to local corruptions. In Robust Face Recognition via Multi-Scale Patch-Based Matrix Regression, Guangwei Gao and Xiao-Yuan Jing show that matrix-based models can identify faces even when they are partially obscured by sunglasses or scarves.
Similarly, Xiao-Jun Wu 0001 and Xing Liu developed 2DR1-PCA and 2DL1-PCA, which apply robust norms (like L1) directly to 2D matrices. By avoiding vectorization, these algorithms maintain the "structural integrity" of the image, allowing them to distinguish between actual features and outliers like bad lighting or sensor noise.
Applications Across Domains
The shift from vectors to tensors is not limited to deep learning; it is transforming classical statistical methods as well:
- Clustering: Chong Peng 0001 and Zhao Kang 0001 show in Subspace Clustering via Variance Regularized Ridge Regression that 2D regularization outperforms vector-based clustering by leveraging the physical arrangement of pixels.
- Probabilistic Modeling: Liqing Zhang and Andrzej Cichocki introduced Tensor-GP, a Gaussian Process framework that uses information divergence to measure similarity between multidimensional structured data without flattening.
- Remote Sensing: For Synthetic Aperture Radar (SAR) imagery, Wei Ye demonstrates in Fast DDL Classification that maintaining the matrix form is essential for real-time, high-accuracy target recognition.
By treating the matrix or tensor as the fundamental unit of information, these researchers have moved away from the "bag of pixels" view of computer vision toward a framework that respects the native geometry of the world.