Abstract
In this work, we empirically show that updating pretrained LMs (350M, 1.3B, 2.7B) with just a few steps of Gradient Ascent Post-training (GAP) on random, unlabeled text corpora enhances its zero-shot generalization capabilities across diverse NLP tasks. Specifically, we show that GAP can allow LMs to become comparable to 2-3x times larger LMs across 12 different NLP tasks. We also show that applying GAP on out-of-distribution corpora leads to the most reliable performance improvements. Our findings indicate that GAP can be a promising method for improving the generalization capability of LMs without any task-specific fine-tuning.
Introduction
Large Language Models (LLMs) are traditionally trained by minimizing prediction error on vast datasets. While effective, this process often leads models to converge into "sharp" local minima—regions of the loss landscape that are deep but narrow. Such sharp minima are frequently associated with poor generalization on unseen tasks, as the model over-optimizes for specific training patterns.
To address this, we introduce Gradient Ascent Post-training (GAP). Instead of the standard approach of seeking deeper minima, GAP intentionally nudges the model out of these sharp regions by briefly maximizing the language modeling loss on unlabeled text. This counter-intuitive "unlearning" process helps the model move toward flatter regions of the loss landscape, enhancing its zero-shot capabilities without requiring any task-specific labeled data.
Methodology
The GAP procedure is remarkably efficient and simple. Starting with a pretrained model (e.g., OPT), we perform a maximum of 15 steps of gradient ascent on a single, randomly sampled snippet of unlabeled text (approximately 200 tokens). Unlike standard training which increases the likelihood of the text, GAP updates parameters to make the text less likely. This approach is conceptually related to Sharpness-Aware Minimization (SAM) but functions as a standalone post-training step to "break" the model out of overfitted states.
Experimental Setup
We evaluated GAP using OPT models of varying sizes (350M, 1.3B, and 2.7B parameters).
- Unlabeled Data for GAP: We sampled text from The Pile, categorizing it by model familiarity: High (Training Data Extraction Challenge), Medium (Common Crawl), and Low/Out-of-Distribution (Github code).
- Evaluation Benchmarks: The models were tested on 12 zero-shot NLP tasks, including dialogue (e.g., Wizard of Wikipedia, Empathetic Dialogues) and reasoning/classification (e.g., Hellaswag, ARC, PIQA).
Results
Performance Scaling and Dialogue Dominance
GAP-enhanced models demonstrated significant performance gains, particularly in generative dialogue tasks. Notably, a 1.3B parameter model updated with GAP achieved a median F1 score that surpassed the performance of a 6.7B parameter baseline model.
Enhancing Generative Quality
Beyond raw F1 scores, GAP significantly improved the qualitative aspects of model output. We observed substantial increases in MAUVE (a measure of human-likeness) and Diversity (a measure of output variety). For instance, the 350M model's MAUVE score increased from 44.3 to 67.2 after GAP.
Impact of Data Source
The choice of text for the ascent steps proved critical. The most consistent improvements were observed when using Out-of-Distribution (OOD) data, specifically Github code. Because the model is less familiar with this data, the gradient ascent effectively pushes the model away from memorized noise without destroying core linguistic knowledge.
Discussion and Limitations
While GAP provides a "free" performance boost for generative tasks, its impact on rigid classification tasks (like ARC or PIQA) is more volatile, sometimes leading to performance degradation in smaller models. This suggests that "unlearning" carries risks for tasks requiring strict logical consistency. Additionally, the process currently requires a small validation set for early stopping, as performance eventually degrades if too many ascent steps are taken. Future work will explore task-agnostic cues to automate the selection of the optimal stopping point and text snippets.
Summary updated