A Preliminary Agentic Framework for Matrix Deflation

Paimon Goulart, Evangelos E. Papalexakis

Abstract

Can a small team of agents peel a matrix apart, one rank-1 slice at a time? We propose an agentic approach to matrix deflation in which a solver Large Language Model (LLM) generates rank-1 Singular Value Decomposition (SVD) updates and a Vision Language Model (VLM) accepts or rejects each update and decides when to stop, eliminating fixed norm thresholds. Solver stability is improved through in-context learning (ICL) and types of row/column permutations that expose visually coherent structure. We evaluate on Digits ($8{\times}8$), CIFAR-10 ($32{\times}32$ grayscale), and synthetic ($16{\times}16$) matrices with and without Gaussian noise. In the synthetic noisy case, where the true construction rank $k$ is known, numerical deflation provides the noise target and our best agentic configuration differs by only $1.75$ RMSE of the target. For Digits and CIFAR-10, targets are defined by deflating until the Frobenius norm reaches $10\%$ of the original. Across all settings, our agent achieves competitive results, suggesting that fully agentic, threshold-free deflation is a viable alternative to classical numerical algorithms.

Introduction

Matrix deflation is a fundamental technique in data compression and signal processing, typically used to isolate dominant structures by sequentially removing rank-1 components. Traditionally, this process relies on rigid numerical thresholds, such as the Frobenius norm, to determine when the "signal" has been exhausted and only "noise" remains. However, these mathematical cutoffs are often arbitrary and scale-dependent, failing to account for the semantic or visual coherence of the underlying data.

This paper introduces a novel multi-agent framework that replaces these numerical thresholds with visual and logical reasoning. By leveraging the complementary strengths of Large Language Models (LLMs) for mathematical generation and Vision Language Models (VLMs) for structural evaluation, we demonstrate a "threshold-free" approach to matrix decomposition.

Methodology

The core of our approach is a collaborative loop between two specialized agents, preceded by a critical preprocessing step.

Permutation Preprocessing

To facilitate visual reasoning, we apply row and column permutations (e.g., Sort or GroupNTeach-Block) to the input matrix. This reordering exposes latent blocky structures, making the matrix's rank-1 components more visually apparent to a vision model.

The Agentic Loop

The deflation process follows a structured pipeline:

  1. Solver Agent (LLM): Using Gemini 2.5 Flash, the solver generates a rank-1 SVD update. To ensure stability and minimize hallucinations, the LLM produces the left singular vector, right singular vector, and a scalar value rather than a full matrix.
  2. Rank-1 Evaluator (VLM): A Qwen-2-VL model compares the permuted input to the proposed reconstruction. It acts as a quality gate, either accepting the update or triggering a regeneration.
  3. Deflation Evaluator (VLM): This agent inspects the residual matrix after subtraction. It serves as the stopping mechanism, terminating the loop only when meaningful visual structure is no longer detectable.
The multi-agent framework for matrix deflation, illustrating the interaction between the LLM solver and VLM evaluators.

Experimental Setup

We evaluated the framework on three distinct datasets, all quantized to 8-bit [0, 255] values:

  • Synthetic (16x16): Matrices with known construction ranks ($k \in [1, 10]$) and Gaussian noise.
  • Digits (8x8): Grayscale handwritten digit images.
  • CIFAR-10 (32x32): Grayscale natural images.

Results and Discussion

Our results indicate that visual evaluation can effectively replace rigid mathematical thresholds. On synthetic data, the agentic approach closely tracked numerical targets, with the best configuration achieving a residual RMSE difference of only 1.75.

Quantitative results across datasets, comparing different permutation strategies and in-context learning (ICL) settings against numerical baselines.

For natural images (CIFAR-10), permutations proved essential. Without reordering, the VLM perceived the matrices as pure noise, leading to failure. However, with the "Sort" permutation, the agent achieved a 31.54 RMSE gap from the numerical target, stopping in fewer steps on average (2.39 vs. 3.05). Interestingly, more complex permutations like GroupNTeach-Block led to more conservative updates, significantly increasing the number of steps required for deflation.

Conclusion

This work provides a preliminary proof-of-concept for agentic, threshold-free matrix deflation. While currently limited to small matrices, the framework demonstrates that splitting tasks between mathematical reasoning (LLM) and structural validation (VLM) creates a stable decomposition loop. Future research will focus on scaling to larger matrices, fine-tuning agents for specific data types, and optimizing permutation strategies to further enhance visual interpretability.

Summary updated

Abstract

Can a small team of agents peel a matrix apart, one rank-1 slice at a time? We propose an agentic approach to matrix deflation in which a solver Large Language Model (LLM) generates rank-1 Singular Value Decomposition (SVD) updates and a Vision Language Model (VLM) accepts or rejects each update and decides when to stop, eliminating fixed norm thresholds. Solver stability is improved through in-context learning (ICL) and types of row/column permutations that expose visually coherent structure. We evaluate on Digits ($8{\times}8$), CIFAR-10 ($32{\times}32$ grayscale), and synthetic ($16{\times}16$) matrices with and without Gaussian noise. In the synthetic noisy case, where the true construction rank $k$ is known, numerical deflation provides the noise target and our best agentic configuration differs by only $1.75$ RMSE of the target. For Digits and CIFAR-10, targets are defined by deflating until the Frobenius norm reaches $10\%$ of the original. Across all settings, our agent achieves competitive results, suggesting that fully agentic, threshold-free deflation is a viable alternative to classical numerical algorithms.
BibTeX
@inproceedings{goulart2026preliminary,
  author = {Paimon Goulart and Evangelos E. Papalexakis},
  title = {{A Preliminary Agentic Framework for Matrix Deflation}},
  booktitle = {WSDM},
  year = {2026},
  doi = {10.1145/3773966.3785506},
}
Figure