Abstract
Standard fine-tuning of pre-trained models often suffers from "neuron bias," where units remain conditioned on source domain features, hindering adaptation to target domains with distinct linguistic patterns. This paper proposes PretRand, an architecture that augments pre-trained networks with a parallel branch of randomly initialized units. By employing independent normalization and learnable weighting, PretRand enables the model to capture domain-specific nuances—such as informal social media syntax—that pre-trained units typically overlook. Evaluated on Twitter Part-of-Speech (POS) tagging, PretRand achieves state-of-the-art results, outperforming standard fine-tuning by an average of 1.4% with minimal parameter overhead.
Introduction
Transfer learning typically involves fine-tuning a model pre-trained on a large source dataset (e.g., formal news) for a smaller target dataset (e.g., Tweets). However, pre-trained neurons often exhibit a "stubborn" adherence to their original training. For instance, a neuron trained on the Wall Street Journal might learn that capitalization signifies a Proper Noun. In the informal context of social media, where capitalization is often used for emphasis or omitted entirely, this bias leads to systematic errors. The authors identify this as a fundamental flaw in standard transfer learning: pre-trained units struggle to "unlearn" source-specific habits and fail to adapt to uncommon target-specific patterns.
The PretRand Architecture
To address neuron bias, the authors introduce the PretRand (Pretrained and Random) framework. The core idea is to provide the model with "blank slate" units that can specialize in target-domain features without the baggage of pre-training.
The architecture consists of several key components:
- Dual-Branch Processing: Input is processed simultaneously through a pre-trained branch and a randomly initialized branch.
- Independent Normalization ($\ell_p$-norm): To prevent the pre-trained branch from dominating the gradient flow early in training, both branches are normalized independently. This ensures the random units receive sufficient signal to learn.
- Learnable Weighting Vectors: Per-class weighting vectors ($u$ and $v$) act as gates, allowing the model to dynamically balance contributions from the pre-trained and random branches based on the specific task requirements.
- Random++ Strategy: A two-stage training process where random units are first optimized while pre-trained weights are frozen, followed by joint fine-tuning.
Experimental Evaluation
The method was tested on Part-of-Speech (POS) tagging across three Twitter-specific datasets: TPoS, ARK, and TweeBank. These datasets represent a significant domain shift from the formal English of the Penn Treebank (WSJ) used for pre-training.
Results and Analysis
PretRand achieved state-of-the-art performance across all tested datasets. Analysis confirmed that while pre-trained units remain highly correlated with their initial states, the random units successfully specialize in domain-specific tokens. For example, a random unit was found to specifically identify the "na" in "gonna" as a particle—a pattern missed by the pre-trained branch.
As shown in the figure above, PretRand consistently outperforms standard fine-tuning, particularly as the amount of target-domain data increases. Notably, the architecture only increases parameter complexity by approximately 2%, making it significantly more efficient than standard ensemble methods which double the parameter count for similar performance gains.
Conclusion
PretRand offers a robust solution to the problem of neuron bias in domain adaptation. By integrating a parallel random branch with careful normalization and weighting, the model retains the benefits of large-scale pre-training while gaining the flexibility to learn target-specific nuances. Future work may explore extending this dual-branch approach to other NLP tasks and deeper architectural layers.
Summary updated