A Preliminary Agentic Framework for Matrix Deflation

Paimon Goulart, Evangelos E. Papalexakis

TL;DR

This paper introduces a novel multi-agent framework for matrix deflation that replaces traditional numerical thresholds with visual and logical reasoning. By pairing a Large Language Model (LLM) to propose rank-1 Singular Value Decomposition (SVD) updates with a Vision Language Model (VLM) to visually evaluate and decide when to stop, the system successfully "peels" apart matrices. This threshold-free approach achieves competitive results against classical numerical methods, opening the door for more structurally-aware scientific data reduction.

What Problem Does This Paper Address?

In scientific data analysis, matrix deflation is a common technique used to isolate signals and compress data. It works by sequentially "peeling away" dominant rank-1 structures (like in SVD or PCA) until only noise remains.

The challenge lies in knowing when to stop peeling. Classical deflation algorithms rely on rigid numerical cutoffs, such as the Frobenius norm. However, these thresholds are scale-dependent and often fail to capture true structural or visual similarity. The authors tackle this gap by asking: Can a team of AI foundation models collaboratively decompose a matrix by visually and logically assessing its meaningful structure, entirely removing the need for arbitrary mathematical thresholds?

Methods

The authors propose a two-agent loop consisting of a "Solver" and an "Evaluator," augmented by a clever preprocessing step:

  • Permutation Preprocessing: Before evaluation, the matrix rows and columns are reordered (using algorithms like Sort or GroupNTeach-Block). This exposes latent, blocky structures, making the matrix visually closer to its rank-1 approximation and much easier for a vision model to evaluate.
  • Solver (LLM - Gemini 2.5 Flash): Instead of generating a massive full matrix, the LLM acts as a math engine. Using In-Context Learning (ICL), it proposes a rank-1 SVD update by generating just the left singular vector, right singular vector, and a scalar value. This reduces hallucinations.
  • Evaluators (VLM - Qwen-2-VL): The VLM acts as both a quality gate and a stopping mechanism.
  • SVD Evaluator: Compares the permuted input to the LLM's reconstructed rank-1 proposal, either accepting it or triggering a regeneration.
  • Deflation Evaluator: Inspects the original and current matrices after the update is subtracted. It stops the loop only when meaningful visual structure is gone, acting as a threshold-free stopping rule.

Datasets

The framework was evaluated on three datasets, all quantized to 8-bit [0, 255] values:

  • Synthetic (16x16): Matrices with known construction ranks ($k$ between 1 and 10) and added Gaussian noise. This served as a ground-truth benchmark where the exact numerical noise target was known.
  • Digits (8x8): Grayscale handwritten digits.
  • CIFAR-10 (32x32): Grayscale versions of natural images.

Key Insights and Ideas

  • Visual Evaluation Beats Rigid Math: VLMs can effectively judge when a matrix has been sufficiently deflated by looking at its visual structure. This proves that semantic, visual stopping rules can successfully replace scale-dependent mathematical thresholds.
  • Permutations are Essential for Natural Images: For complex, natural data like CIFAR-10, unpermuted matrices looked like pure noise to the VLM, leading to repeatedly rejected proposals. Reordering rows and columns to reveal blocky structures was absolutely mandatory for the VLM to "see" the patterns.
  • Agentic Collaboration Works for Linear Algebra: Splitting the task—having an LLM do the heavy mathematical reasoning (proposing vectors) and a VLM do the structural validation—creates a surprisingly stable and effective decomposition loop without requiring model fine-tuning.

Key Results

The agentic framework achieved results highly competitive with classical numerical baselines (which deflated until the residual norm reached 10% of the original):

  • Synthetic Data: The agentic approach closely tracked the numerical target. The best configuration (no permutation, 2 ICL examples) missed the target residual RMSE by only 1.75.
  • Digits: Using a simple "Sort" permutation yielded the best results, with a 16.43 RMSE gap from the numerical target. It took an average of 3.75 steps to stop, compared to the baseline's 4.2 steps.
  • CIFAR-10: The "Sort" permutation again won out, achieving a 31.54 RMSE gap and stopping in 2.39 steps (vs. 3.05 numerical steps). Unpermuted matrices completely failed on this dataset.
  • Algorithmic Behavior: Using the GroupNTeach-Block permutation led to highly conservative subtractions, requiring significantly more steps to deflate (e.g., 10.52 steps on synthetic data vs. the expected 5.5).

Limitations and Future Work

Limitations: The current framework is a preliminary proof-of-concept tested only on very small matrices (up to 32x32). Additionally, certain permutation strategies (like GroupNTeach-Block) cause the agents to make overly conservative updates, increasing the computational cost by requiring more deflation steps.

Future Work: The authors plan to scale this approach to larger datasets. They also aim to fine-tune the agents specifically for this task, incorporate In-Context Learning directly into the VLM evaluator's prompt, and explore new permutation strategies to optimize the visual structure presented to the models.

Summary updated

Abstract

Can a small team of agents peel a matrix apart, one rank-1 slice at a time? We propose an agentic approach to matrix deflation in which a solver Large Language Model (LLM) generates rank-1 Singular Value Decomposition (SVD) updates and a Vision Language Model (VLM) accepts or rejects each update and decides when to stop, eliminating fixed norm thresholds. Solver stability is improved through in-context learning (ICL) and types of row/column permutations that expose visually coherent structure. We evaluate on Digits ($8{\times}8$), CIFAR-10 ($32{\times}32$ grayscale), and synthetic ($16{\times}16$) matrices with and without Gaussian noise. In the synthetic noisy case, where the true construction rank $k$ is known, numerical deflation provides the noise target and our best agentic configuration differs by only $1.75$ RMSE of the target. For Digits and CIFAR-10, targets are defined by deflating until the Frobenius norm reaches $10\%$ of the original. Across all settings, our agent achieves competitive results, suggesting that fully agentic, threshold-free deflation is a viable alternative to classical numerical algorithms.
BibTeX
@inproceedings{goulart2026preliminary,
  author = {Paimon Goulart and Evangelos E. Papalexakis},
  title = {{A Preliminary Agentic Framework for Matrix Deflation}},
  booktitle = {WSDM},
  year = {2026},
  doi = {10.1145/3773966.3785506},
}
Figure