Paolo Glorioso

1 previous affiliation
2Machine learning papers tracked by Lacuna
Is this you?

Research Overview

Paolo Glorioso focuses on the optimization of overparameterized neural networks and the structural efficiency of large language models. His early work investigated the implicit bias of stochastic gradient descent and principled approaches to hyperparameter scaling. More recently, his research addresses architectural redundancy in deep transformers, exploring methods to prune deep layers while preserving model performance.

Research Trajectory

Glorioso's research trajectory begins with foundational investigations into the optimization dynamics of deep learning models. In 2023, his work addressed the trade-offs between generalization and training speed in commonly used optimization algorithms. Specifically, he guided research on the implicit bias that arises from the interplay between stochastic gradient descent (SGD) with label noise and momentum in overparameterized neural networks. This effort culminated in Flatter, Faster: Scaling Momentum for Optimal Speedup of SGD, which introduced a principled method for selecting the momentum hyperparameter. By scaling this parameter with the learning rate to the power of two-thirds, the approach maximally accelerates training while maintaining the strong generalization typically associated with SGD.

Following this work on optimization, Glorioso's focus expanded to the structural efficiency and architectural properties of large language models. In 2025, he participated in a study examining the necessity of deep layers in popular architectures such as Llama-2-70B. The resulting paper, The Unreasonable Ineffectiveness of the Deeper Layers, demonstrated that up to half of a model's deepest layers can be removed with almost no degradation in question-answering performance. This research highlighted that factual knowledge is primarily localized in the shallow layers of the network. To restore any lost capabilities, the authors applied a small amount of parameter-efficient fine-tuning to "heal" the pruned network. This contribution advances several interconnected research areas, including the direction of Depth Redundancy in Large Language Models and the broader area of Functional Redundancy and Layer Collapse in Deep Transformers. Furthermore, it provides practical insights for the research direction of Performance Recovery in Pruned Neural Networks.

Current Directions

The most recent papers in the evidence indicate a strong focus on the structural efficiency of large language models. Current work investigates the extent to which deep transformer architectures exhibit functional redundancy, particularly questioning the utility of deeper layers for factual recall and reasoning. By exploring techniques to structurally prune massive networks and subsequently heal them through parameter-efficient fine-tuning, this research aims to drastically reduce the computational footprint of state-of-the-art models without sacrificing their core capabilities.

Profile updated

See something wrong? Report a mistake