Analyzing how a massive neural network "thinks" usually requires an equally massive amount of memory. If you want to know which training examples influenced a specific prediction (Training Data Attribution) or understand the geometry of the loss landscape (Hessian analysis), you often need to store gradients—vectors that are as large as the model itself—for thousands or millions of data points. For a model with billions of parameters, researchers quickly hit a "memory wall."
In a new paper, Andrea Schioppa introduces a suite of "sketching" algorithms designed to break through this wall. These methods allow researchers to study large-scale models on modern hardware like GPUs and TPUs with unprecedented efficiency.
The Scalability Challenge
To truly understand a model's behavior, researchers typically investigate three key areas:
- Training Data Attribution (TDA): Identifying which specific training document caused the model to generate a particular output.
- Intrinsic Dimension (ID): Determining the minimum number of parameters actually needed to solve a given task.
- Hessian Analysis: Mapping the "terrain" of the loss function to see if it consists of steep valleys or flat plains.
All of these tasks require storing or repeatedly calculating high-dimensional vectors. Traditional methods either use "Dense Projections," which require too much memory, or "Layer Selection," which only looks at a few layers to save space. Furthermore, existing efficient transforms like the Fastfood Transform (FFD) perform poorly on modern hardware like TPUs because they rely on slow memory lookups.
Hardware-Optimized Sketching
To solve these bottlenecks, the paper proposes a framework for Scalable Gradient and HVP (Hessian Vector Product) Sketching. The key innovation is moving away from memory-heavy lookups and toward hardware-friendly operations.
- New Algorithms: The author introduces AFFD (Augmented Fastfood), AFJL (Augmented Fast Johnson-Lindenstrauss), and QK (a Kronecker-based approach).
- Kronecker Products: Instead of using one giant random matrix to compress data, these methods use the Kronecker product of several smaller matrices. This allows the hardware to perform compression using fast, local matrix multiplications rather than jumping around in memory.
The paper also explores "Explicit" sketching (projecting the gradient directly) versus "Implicit" sketching (perturbing the model weights and measuring the change in loss). Surprisingly, explicit sketching was found to be significantly faster, reducing wall-time by about 70%. Unlike previous heuristic approaches, this paper provides formal mathematical proofs to guarantee that these compressed "sketches" accurately preserve the geometric properties of the original high-dimensional data.
Key Insights & Discoveries
By applying these efficient sketches to models like GPT-2, RoBERTa, and BART, the research uncovered several critical insights about LLM behavior:
- Layer Selection is a Trap: Many researchers try to save memory by only calculating TDA scores for the last few layers of a model. This paper proves that this introduces substantial distortion. To get an accurate picture, you need to analyze the whole model, which sketching makes computationally feasible.
- Generative Tasks are Complex: Previous studies on smaller models suggested that neural networks have a very low "intrinsic dimension" (meaning they only use a tiny fraction of their parameters). This paper finds the opposite for generative LLMs: their intrinsic dimension can approach their full parameter count, suggesting generative tasks are vastly harder than simple classification.
- LLM Hessians are Different: The "Hessian Spectrum" (a map of the loss landscape's curvature) in LLMs doesn't follow the same rules as smaller networks. For instance, the outlier eigenvalues that usually dominate the landscape behave differently in models with layer normalization, challenging long-held assumptions in deep learning theory.
Performance Results
The practical benefits of these new algorithms are substantial:
- Speed and Efficiency: On TPUs, the new AFJL algorithm achieved a 100x wall-time reduction compared to traditional FJL. On GPUs, the QK algorithm reduced peak memory usage by over 4x compared to standard methods.
- Constant Scaling: Unlike previous methods where computation time increases as you demand higher target dimensions, the proposed methods exhibit almost constant wall-time. This makes them highly predictable for large-scale engineering.
- High Fidelity: Despite the massive compression, the sketches achieved a 98% correlation with the uncompressed "ground truth" data.
Limitations and Future Work
While the paper provides a powerful new toolkit, there are areas for future exploration. The benchmarks focused heavily on Transformers, so performance might vary on other architectures like State Space Models (SSMs) or CNNs. Additionally, the finding that generative tasks have high intrinsic dimensions suggests that linear sketching might eventually hit a limit, pointing toward non-linear projection algorithms as a promising next step for finding even more compact representations.
Summary updated