Qiaosen Wang

University of Chicago
2 previous affiliations
3Machine learning papers tracked by Lacuna
Is this you?

Research Overview

Qiaosen Wang focuses on the optimization dynamics and parameter-efficient adaptation of large-scale neural networks. Wang’s research explores the intersection of theoretical optimization—specifically the behavior of stochastic gradient descent in normalized models—and the practical engineering of fine-tuning methodologies for large language models. This work aims to improve the efficiency of model adaptation by developing techniques for dynamic rank allocation and making computationally intensive second-order optimization methods viable for billion-parameter architectures.

Research Trajectory

Wang’s research trajectory is characterized by a simultaneous focus on the mathematical foundations of deep learning and the applied challenges of the current large-model paradigm. The available evidence from 2023 indicates a concentrated effort to address the "weight problem" of modern architectures through two primary lenses: the refinement of optimization algorithms and the advancement of low-rank adaptation techniques.

In the realm of optimization theory, Wang has investigated how fundamental components of modern training pipelines, such as normalization and weight decay, influence the trajectory of stochastic gradient descent. By analyzing simplified mathematical models, this work seeks to provide a more rigorous understanding of the "Spherical Motion Dynamics" that govern how weights evolve on a hypersphere during training. This theoretical grounding informs Wang’s more applied work in parameter-efficient fine-tuning (PEFT), where the goal is to achieve high performance with minimal updates to the underlying model parameters.

Wang’s contributions to PEFT involve moving beyond static adaptation methods. While early approaches to low-rank adaptation relied on uniform configurations across all model layers, Wang’s work has moved toward more flexible, sparse, and dynamic structures. This evolution reflects a broader shift in the field toward resource-aware machine learning, where the complexity of the adaptation mechanism is scaled according to the specific needs of the task and the individual layers of the transformer architecture.

Current Directions

A central theme in Wang’s current research is Adaptive Rank Allocation for Low Rank Adaptation. This direction addresses the inefficiency of assigning the same rank to every layer in a network, regardless of that layer's functional importance. By developing methods for Dynamic Rank Allocation for Parameter Efficient Fine Tuning, Wang’s work allows for a more nuanced distribution of trainable parameters, which can lead to better performance-to-parameter ratios.

This focus on efficiency extends to Automated Rank Selection for Low-Rank Neural Network Adaptation, where the objective is to automate the search for optimal low-rank structures. Such automation is vital for scaling fine-tuning processes across diverse hardware constraints and task requirements. Furthermore, Wang’s work intersects with broader systemic challenges in distributed learning, such as Dynamic Gradient Sparsification for Distributed Learning, which aims to alleviate communication bottlenecks by reducing the volume of data exchanged between nodes during training.

Additionally, the research touches upon the qualitative aspects of model behavior during fine-tuning, including the challenge of Mitigating the Alignment Tax in Large Language Models. As models are aligned with human preferences or safety constraints, they often suffer a decline in general capabilities; Wang’s work in efficient and sparse adaptation provides a potential framework for minimizing these trade-offs by isolating and protecting core model functions during the alignment process.

Representative Work

In Applying Second Order Optimization to Deep Transformers with Parameter-Efficient Tuning, Wang (as second author) addresses the long-standing issue of the high computational and memory costs of second-order optimizers. While these optimizers, such as those based on the Hessian matrix or its approximations, are theoretically superior in terms of convergence speed and their ability to navigate complex loss landscapes, they are rarely used for large-scale models due to the quadratic scaling of memory requirements. Wang and colleagues demonstrate that the Parameter-Efficient Fine-Tuning (PEFT) paradigm—where only a tiny fraction of parameters are optimized—fundamentally changes this trade-off. By restricting the second-order optimization to this small subset of parameters, the computational overhead becomes manageable even for models with hundreds of millions to billions of parameters. This work verifies the tractability of these methods and suggests that the efficiency gains of PEFT can be leveraged not just to save memory, but to enable more sophisticated optimization strategies that were previously computationally prohibitive.

Wang’s theoretical contributions are exemplified in How Normalization and Weight Decay Can Affect SGD? Insights from a Simple Normalized Model, where he served as second author. The paper investigates the "Spherical Motion Dynamics" (SMD) of normalized models trained with Stochastic Gradient Descent (SGD) and weight decay (WD). Normalization layers, such as Batch Normalization or Layer Normalization, are standard in modern architectures, but their interaction with weight decay and SGD is complex. By introducing and analyzing the Noisy Rayleigh Quotient (NRQ) model—a simplified mathematical framework—the research provides a theoretical proof that SMD can dominate the training process. Specifically, it shows how these factors control the evolution of "angular updates," which represent the change in the direction of the weight vector rather than its magnitude. This analysis offers critical insights into how the interaction of common training hyperparameters affects the geometric properties of the optimization path and the final performance of the model.

Wang also participated in research on Sparse Low-rank Adaptation of Pre-trained Language Models. This work introduces Sparse Low-rank Adaptation (SoRA), which improves upon the standard Low-Rank Adaptation (LoRA) framework. The original LoRA methodology is based on the hypothesis that the change in weights during model adaptation has a low "intrinsic rank." However, standard LoRA implementations use a fixed, pre-defined rank for all layers. SoRA extends this by allowing for dynamic adjustments to the intrinsic rank during the adaptation process. By using a sparse approach, the model can effectively reduce the rank of adaptation matrices in layers where they are less necessary, while maintaining higher capacity in more critical layers. This research highlights Wang’s involvement in developing next-generation PEFT methods that prioritize both flexibility and parameter efficiency, ensuring that large-scale language models can be fine-tuned effectively even under strict resource constraints.

Profile updated

See something wrong? Report a mistake