Learning Directly on Multidimensional Tensor Structures

376 concepts325 papers1 proposal

When we work with multidimensional data—such as a 3D medical fMRI scan, a video sequence, or a hyperspectral image cube—our standard machine learning toolkit often demands a destructive first step: flattening the data. Whether we unfold a 3D tensor into a 2D matrix or stretch it into a 1D vector, this process of vectorization or matricization fundamentally alters the data's geometry.

Ripping apart the multi-way grid structure forces variables that were once spatially, temporally, or spectrally adjacent to be treated as independent entities. A rich body of research focuses on eliminating this structural information loss by designing algorithms that operate directly on native tensor structures.

The Cost of Flattening: Why Geometry Matters

To understand why tensor-native learning works, we must first look at what goes wrong when we flatten a tensor. Consider a 3D video tensor with dimensions of height, width, and time. In its native form, a pixel at coordinate $(x, y, t)$ shares an intimate structural relationship with its neighbors in all three directions.

As Qiquan Shi et al. point out, converting these multidimensional tensors into one-dimensional vectors destroys these inherent spatial and structural relationships. When the data is flattened, the model loses the concept of topological proximity. The algorithm no longer "knows" that two features were adjacent in physical space or consecutive in time; it must attempt to relearn these dependencies from scratch.

Beyond losing geometric context, flattening triggers severe computational challenges:

  1. The Curse of Dimensionality: Vectorizing high-dimensional inputs creates massive feature vectors. As Lifang He et al. observe, this flattening process triggers the curse of dimensionality, making it exceptionally difficult to learn effectively from high-dimensional inputs when sample sizes are small.
  2. Parameter Explosion: Standard linear layers or classifiers operating on flattened vectors require a unique weight for every single element combination. Qiquan Shi et al. note that this vectorization process creates models with an excessive number of parameters, leading to high computational demands and a severe risk of overfitting.
  3. Loss of Multi-Way Interactions: A standard matrix can only capture two-way interactions (rows and columns). As Brett W. Bader explains, representing multi-way data as two-dimensional matrices requires a flattening process (matricization) that destroys the structural information and subtle dependencies inherent in the original high-order data.

The Solution: Mode-Wise Operations and Tensor Decompositions

Instead of forcing the data to fit matrix-based algorithms, tensor-native approaches reshape the algorithms to fit the data. They achieve this by leveraging multilinear algebra, where operations are performed "mode-wise"—meaning they interact with one dimension of the tensor at a time while preserving the overall multi-way structure.

Preserving Subspaces via Tensor Factorization

Rather than learning a massive, unstructured weight matrix, tensor-native models factorize weights or features into low-rank components that mirror the dimensions of the input data.

For example, in regression tasks, traditional methods flatten the input features and learn a flat vector of regression coefficients. This makes it incredibly difficult to interpret the physical significance of the coefficients. To solve this, Qibin Zhao et al. introduced Higher-Order Partial Least Squares (HOPLS). By avoiding unfolding, HOPLS leverages the natural organization of the input features, making the resulting regression coefficients physically interpretable and structurally cohesive.

Similarly, when dealing with missing data, traditional matrix completion techniques require unfolding tensors into matrices. As Zemin Zhang et al. argue, this ignores the rich structural relationships between dimensions. By operating directly on the tensor using frameworks like the tensor Singular Value Decomposition (t-SVD), models can capture complex lateral structures and dependencies that standard matrix completion completely misses (as detailed by Michael Ng et al.).

Flattening Approach (Destructive):
[3D Tensor] ---> [Unfolding/Vectorization] ---> [Flat 1D/2D Vector] ---> [Loss of Spatial/Temporal Proximity]

Tensor-Native Approach (Structure-Preserving):
[3D Tensor] ---> [Mode-1 Operations] x [Mode-2 Operations] x [Mode-3 Operations] ---> [Maintained Geometry]

How Tensor-Native Learning Works in Practice

By maintaining the tensor structure, researchers have successfully redesigned core machine learning primitives—including optimization, kernels, and attention mechanisms.

Structure-Aware Optimization

In deep learning, conventional optimizers like Adam or standard SGD flatten parameter tensors into long vectors before computing updates. This treats every weight independently, ignoring the fact that the weights belong to structured layers (like convolutional kernels).

To exploit this structure, Vineet Gupta et al. developed Shampoo, a structure-aware preconditioning algorithm. Instead of maintaining a massive, flat preconditioner for the entire vectorized parameter space, Shampoo treats model parameters as tensors and maintains separate, smaller preconditioners for each individual dimension. This captures the structural relationships across neural network layers while drastically reducing computational overhead.

Tensor Kernels and Classifiers

For supervised learning, standard kernel methods (like those used in Support Vector Machines) require inputs to be flat vectors. Ming Hou et al. note that this requirement discards the multi-way correlations essential for capturing spatial and temporal dependencies.

To bridge this gap, Lifang He et al. designed the Kernelized Support Tensor Machine (KSTM). KSTM jointly optimizes a kernelized tensor factorization alongside a maximum-margin classifier. Because it operates directly on the tensor, it successfully captures complex, non-linear relationships without succumbing to the parameter explosion that plagues vector-based kernels.

Factorized Attention for Higher-Order Data

The rise of Transformers has made sequence-based processing ubiquitous, but converting multidimensional data like 3D medical scans or videos into 1D sequences removes critical structural context.

To address this limitation, Soroush Omranpour et al. introduced the Higher-Order Transformer (HOT). HOT replaces standard flat attention with a factorized attention mechanism represented as a sum or Kronecker product of mode-wise matrices. This allows the model to capture complex cross-dimensional dependencies with significantly lower computational and memory costs, proving that even attention mechanisms can be redesigned to respect native tensor geometries.

Why It Works: The Intuition of Inductive Bias

Ultimately, tensor-native learning works because it introduces a powerful and accurate inductive bias. By forcing the model's internal mathematical operations to match the multi-way grid structure of the data, we prevent the model from exploring physically impossible parameter spaces.

As Yada Zhu demonstrates, preserving the structural integrity of the data and incorporating domain knowledge into a low-rank weight tensor significantly outperforms flattening techniques, particularly by preventing overfitting when labeled data is scarce. When the mathematics of the model respects the geometry of the world, the model requires fewer parameters, trains faster, and generalizes far better to unseen data.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 296 papers
  2. Version 1 · 2026-10-04 · 469 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_structural_information_loss_tensor_flatt_37c513b6,
  title = {structural information loss (tensor flattening artifacts)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-information-loss-tensor-flattening-artifacts/rdr_37c513b6a39b410eb3f4772bd0846daf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_structural_information_loss_tensor_flatt_37c513b6_v2,
  title = {structural information loss (tensor flattening artifacts)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-information-loss-tensor-flattening-artifacts/rdr_37c513b6a39b410eb3f4772bd0846daf/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake