Hierarchical Representation Learning via Deep Matrix Factorization

36 concepts30 papers1 proposal

In unsupervised machine learning, matrix factorization is a foundational tool for discovering latent structure. By decomposing an observation matrix $X$ into a basis matrix $W$ and a coefficient matrix $H$ ($X \approx WH$), traditional techniques like Non-negative Matrix Factorization (NMF) or Singular Value Decomposition (SVD) map high-dimensional data into a single, flat latent space.

However, real-world data is rarely flat. In text corpora, words combine into narrow themes, which in turn form broad topics. In computer vision, pixels form edges, edges form object parts, and parts form entire objects. Standard single-layer matrix factorizations force these multi-scale, nested dependencies into a single layer of abstraction.

To capture this inherent hierarchy, researchers have turned to Deep Matrix Factorization (DMF). By stacking multiple factorization layers, DMF mirrors the hierarchical feature extraction of deep neural networks while preserving the explicit, parts-based interpretability of matrix decomposition.

How Stacking Builds Abstraction

The core intuition behind deep matrix factorization is straightforward: the latent representation produced by one layer serves as the input for the next.

Mathematically, instead of a single decomposition, a multi-layer architecture factorizes the data matrix through a chain of successive sub-layers:

$$X \approx W_1 H_1$$ $$H_1 \approx W_2 H_2$$ $$\dots$$ $$H_{m-1} \approx W_m H_m$$

Substituting these layers yields a global reconstruction formula: $X \approx W_1 W_2 \dots W_m H_m$.

In this setup, each matrix $W_i$ acts as a mapping between different levels of abstraction. As demonstrated by Hyun Ah Song et al. in Hierarchical Data Representation Model - Multi-layer NMF, this progressive aggregation allows low-level features to naturally group into increasingly abstract, high-level concepts.

[ Raw Data X ] 
      │
      ▼  (Factorization Layer 1: Captures low-level features, e.g., edges)
 [ W1 ] × [ H1 ]
            │
            ▼  (Factorization Layer 2: Groups features into parts)
       [ W2 ] × [ H2 ]
                  │
                  ▼  (Factorization Layer 3: Aggregates parts into concepts)
             [ W3 ] × [ H3 ]

This phenomenon is highly visible in image processing. Guoxu Zhou et al. showed in Learning the Hierarchical Parts of Objects by Deep Non-Smooth Nonnegative Matrix Factorization that early layers capture localized, basic edges, while deeper layers assemble these edges into complex structural components like facial organs.

Even simple linear operations benefit from this hierarchical stacking. For instance, Hinrich Schuetze et al. discovered in Two SVDs produce more focal deep learning representations that applying SVD twice with an intermediate normalization step creates "focal" representations where individual dimensions are significantly more independent and semantically meaningful than those generated by a single-layer reduction.

The Multi-Layer Filter: Noise Reduction and Subspace Discovery

Beyond building hierarchies, stacking factorization layers serves a critical regularizing function: it acts as a multi-stage noise filter.

In high-dimensional datasets, observations are often corrupted by noise or view-specific idiosyncrasies. Because each layer in a deep matrix factorization forces the data through a low-rank bottleneck, it strips away superficial variations.

This mechanism is particularly powerful in multi-view learning, where the goal is to find a shared identity across different data sources (such as text descriptions and image features of the same object). As Handong Zhao et al. demonstrated in Multi-View Clustering via Deep Matrix Factorization, deep semi-NMF architectures progressively isolate core semantic features from view-specific noise by filtering the representations layer by layer.

Similarly, Xianzhen Li et al. leveraged this property in Multilayer Collaborative Low-Rank Coding Network for Robust Deep Subspace Discovery, showing that a multi-layer framework refines raw data into increasingly salient, low-rank components. By capturing complex, non-linear distributions across multiple steps, deep architectures extract robust latent attributes that shallow models often miss (Zhe Xue et al. in Deep Correlated Predictive Subspace Learning for Incomplete Multi-View Semi-Supervised Classification).

From Greedy Stacking to Joint Optimization

Early attempts to build deep matrix factorizations relied on greedy training: factorizing the raw data first, fixing the first layer's matrices, and then factorizing the resulting coefficient matrix. However, this feed-forward approach suffers from severe error propagation—any reconstruction error introduced in the first layer compounds downstream.

To solve this, modern frameworks treat the entire stack as a joint optimization problem. As reviewed by Zhao Zhang et al. in A Survey on Concept Factorization: From Shallow to Deep Representation Learning, updating basis concepts and representations simultaneously across all layers prevents error accumulation and yields much more stable data representations.

One elegant way to achieve this joint optimization is by bridging the gap between matrix factorization and neural networks through "unfolding." For example, Jonghye Woo et al. in A deep joint sparse non-negative matrix factorization framework for identifying the common and subject-specific functional units of tongue motion during speech unfolded iterative optimization algorithms into deep neural network architectures. This allows the model to capture complex, non-linear physical motions while maintaining strict interpretability.

Taking this a step further, Tyler Will et al. introduced Neural Nonnegative Matrix Factorization for Hierarchical Multilayer Topic Modeling, where each layer's activations are defined as the solution to a constrained optimization problem. By deriving a complete backpropagation scheme, the model refines topics across all layers simultaneously, producing highly accurate and interpretable "topic trees."

Greedy Stacking (Error Propagates):
Layer 1 Optimized ──► (Fixed) ──► Layer 2 Optimized ──► (Fixed) ──► Layer 3 Optimized

Joint/Neural Optimization (Error Minimization via Backprop):
Layer 1 ◄───────────────► Layer 2 ◄───────────────► Layer 3  (Simultaneous Updates)

To ensure these deep architectures converge reliably during global optimization, Nicolas Gillis et al. developed layer-centric and data-centric loss functions in A consistent and flexible framework for deep matrix factorizations, providing the mathematical stability needed to extract clean hierarchies in complex imaging and facial recognition tasks.

Dual Hierarchies and Generative Frameworks

The evolution of deep matrix factorization has expanded beyond simply stacking feature representations. Standard DMF models only decompose the feature matrix hierarchically. To capture even richer structural relationships, Zijian Yang et al. introduced Generalized Discriminative Deep Non-Negative Matrix Factorization Based on Latent Feature and Basis Learning, a framework that simultaneously decomposes both the basis matrix and the feature matrix into multiple hierarchical layers. This dual-hierarchy approach maps out the structural parts and their corresponding weights at the same time.

Finally, the boundaries between matrix factorization and probabilistic generative models are beginning to dissolve. In Multilayer Matrix Factorization via Dimension-Reducing Diffusion Variational Inference, Junbin Liu et al. framed multilayer matrix factorization as the reverse generative process of a diffusion model. Here, each factorization layer reconstructs a higher-dimensional representation from a lower-dimensional, noisy latent variable. By embedding classical matrix decomposition within a modern diffusion framework, this approach demonstrates that the core principles of deep matrix factorization remain vital to cutting-edge representation learning.

Key Foundational Work

A Deep Semi-NMF Model for Learning Hidden Representations (2014) — contributed 2 concept elements to this direction.
Hierarchical Data Representation Model - Multi-layer NMF (2013) — contributed 1 concept elements to this direction.
Learning Multi-level Sparse Representations (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_matrix_factorization_dd9117c5,
  title = {hierarchical matrix factorization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-matrix-factorization/rdr_dd9117c559e8496caa56bc1b4bcb8021},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_matrix_factorization_dd9117c5_v1,
  title = {hierarchical matrix factorization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-matrix-factorization/rdr_dd9117c559e8496caa56bc1b4bcb8021/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake