Can a Large Language Model Learn Matrix Functions In Context?

Paimon Goulart, Evangelos E. Papalexakis

Introduction

While Large Language Models (LLMs) have demonstrated remarkable proficiency in natural language tasks, their capacity for "pure" numerical reasoning remains an active area of investigation. Recent studies have shown that LLMs can perform simple linear regression in-context, learning patterns from a few examples provided in a prompt. However, many critical scientific and engineering challenges require high-dimensional, non-linear computations, such as Singular Value Decomposition (SVD). This paper explores whether LLMs can approximate these complex matrix functions through In-Context Learning (ICL) and how their performance compares to classical machine learning baselines.

Methodology

The researchers utilized a "meta-in-context learning" framework to test the models' numerical intuition. A key feature of this approach is that the models were never explicitly informed of the specific mathematical task they were performing. Instead, they were presented with a generic prompt that framed the problem as observing a "machine" producing an output $y$ for a given input $x$.

The generic prompt structure used to evaluate mathematical functions without explicit instructions.

The study evaluated Google’s Gemini-1.5-flash alongside open-source models like Qwen2.5-72B and Llama-3.1-8B. These were benchmarked against three traditional baselines: SGD-based Linear Regression, a 2-layer Neural Network (NN), and a Convolutional Neural Network (CNN) specifically for matrix-based tasks. The models were tasked with predicting vector $p$-norms, nuclear norms, and top-$k$ singular values using synthetic datasets of vectors and matrices (up to $25 \times 25$) with entries ranging from -100 to 100.

Key Insights

The Complexity Paradox

A significant finding is that while traditional 2-layer NNs can eventually outperform LLMs on simple tasks like vector norms given sufficient data, the LLM advantage grows as the mathematical complexity increases. For non-linear tasks like SVD, LLMs appear more capable of capturing high-level patterns than simple regression models.

Sample Efficiency and Convergence

LLMs exhibit extreme sample efficiency in numerical tasks. Their error rates, measured by Root Mean Square Error (RMSE), typically plateau after observing only approximately 20 examples. In contrast, classical models often require significantly more data to reach comparable accuracy or suffer from overfitting, leading to erratic error spikes as the context window fills.

RMSE convergence for vector norms, illustrating Gemini's sample efficiency compared to traditional baselines.

Scalability and Architecture

As matrix dimensions scaled from $5 \times 5$ to $25 \times 25$, the performance of SGD and 2-layer NN baselines degraded significantly. Gemini, however, maintained stable and competitive accuracy, performing nearly as well as a CNN architecture designed for spatial data. Furthermore, this capability is not limited to proprietary models; open-source models like Qwen and Llama showed similar performance curves, suggesting that numerical ICL is a fundamental property of modern transformer architectures.

Key Results

Quantitative analysis reveals a clear advantage for LLMs in complex scenarios. In predicting the top-3 singular values of a $5 \times 5$ matrix, Gemini achieved an average RMSE of 21.79, outperforming SGD (23.58) and the 2-layer NN (28.87). The performance gap widened dramatically as the problem size increased. For $25 \times 25$ matrices, Gemini's RMSE was 24.23, while SGD and the 2-layer NN collapsed with RMSEs of 241.31 and 334.38, respectively.

Conclusion and Future Work

This research demonstrates that LLMs can serve as effective "black-box" approximators for complex, non-linear matrix functions through in-context learning. They are particularly robust in low-data regimes and scale better than simple neural baselines for high-dimensional tasks.

Future research should address current limitations, such as scaling to larger matrix sizes common in "Big Data" applications and investigating the impact of different numerical representations (e.g., scientific notation) on precision. Additionally, exploring more abstract operations like matrix inversion or Eigenvalue decompositions will help define the boundaries of LLM numerical intuition.

Summary updated

BibTeX
@inproceedings{goulart2024can,
  author = {Paimon Goulart and Evangelos E. Papalexakis},
  title = {{Can a Large Language Model Learn Matrix Functions In Context?}},
  booktitle = {IEEE Big Data},
  year = {2024},
  doi = {10.1109/bigdata62323.2024.10825481},
}

Loading related content...

Figure