Introduction
In modern Natural Language Processing (NLP), pre-training on massive corpora has become the standard "warm-up" before fine-tuning models for specific downstream tasks. While various pre-training objectives—such as masked language modeling or span infilling—often yield comparable scores on standard benchmarks like GLUE or SuperGLUE, this research suggests that these objectives are not interchangeable. Instead, they leave "vestigial behaviors": internal mechanics and biases that dictate how a model responds to control signals long after the pre-training phase has ended.
The Controllability Gap
The investigation began with a puzzling observation regarding Context Augmentation (CtxAug). CtxAug is a zero-shot control method designed to steer a model toward specific attributes (e.g., making a dialogue response more "inquisitive") by prepending attribute representations to the encoder's output. While CtxAug proved highly effective for BART models, it failed significantly when applied to T5, despite both being high-performing encoder-decoder architectures. This discrepancy suggests that the underlying pre-training objectives—BART’s full reconstruction versus T5’s partial reconstruction—fundamentally alter the model's "steerability."
Methodology: Isolating the Variables
To determine why certain models are more controllable than others, the authors conducted controlled experiments using scaled-down BART architectures (approximately 1M parameters). By keeping the data, model size, and fine-tuning process constant, they isolated the pre-training objective as the sole independent variable.
The study compared several objectives:
- BART Default (MLM+PS): Masked Language Modeling combined with Sentence Permutation.
- Span-Infilling with Full Reconstruction (SIFR): The model must reconstruct the entire original sequence from a corrupted version.
- Span-Infilling with Partial Reconstruction (SIPR): The model only reconstructs the missing parts (the T5 approach).
These models were pre-trained on BookCorpus and subsequently fine-tuned on Topical-Chat to generate dialogue responses with specific stylistic attributes: inquisitiveness and sentiment.
Results: The Power of Full Reconstruction
The experimental results highlight a stark contrast in controllability. As shown in Figure 1, BART-base models exhibit a dramatic increase in inquisitive responses when CtxAug is applied. In contrast, T5-style models (T5-small and T5-small-LM) show negligible or even negative responses to the same control signals.
The "secret sauce" for controllability appears to be Full Reconstruction (SIFR). When a model is forced to reconstruct the entire input during pre-training, it develops a stronger coupling between the input context and the generated output. This structural bias makes the model far more sensitive to external "control codes" injected at inference time. Conversely, partial reconstruction (SIPR) creates a "disjointed" relationship, where the model learns to ignore large portions of the context, making it resistant to zero-shot steering.
Vestigial Behaviors and Attribute Sensitivity
The authors term these persistent traits "vestigial behaviors." These are properties that remain after fine-tuning and can be traced directly back to the pre-training objective. While these behaviors might be invisible during standard evaluation, they become critical when attempting to manipulate model behavior.
Interestingly, the ease of control varies by attribute. Sentiment (positive/negative) was relatively easy to control across all models, likely because sentiment features are strongly represented in the fine-tuning data itself. However, controlling for inquisitiveness—a structural linguistic trait—required the specific bias provided by full-reconstruction pre-training.
As illustrated in Figure 6, the $SI_{FR}$ objective significantly enhances the model's ability to increase inquisitive responses compared to a model with no pre-training (No PT). This demonstrates that the pre-training phase provides a "useful bias" that persists throughout fine-tuning.
Conclusion
This research serves as a crucial reminder for model developers: the choice of pre-training objective involves trade-offs beyond simple computational efficiency. While partial reconstruction (like T5) may be faster to train, it may produce models that are harder to control or steer in production environments. For applications requiring high levels of behavioral control, objectives that emphasize full sequence reconstruction, like those used in BART, offer a significant advantage. Future work will need to explore how these dynamics scale to multi-billion parameter models and whether hybrid objectives can balance efficiency with controllability.
Summary updated