Optimizing Deeper Transformers on Small Datasets

Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung, Simon J. D. Prince, Yanshuai Cao

TL;DR

This paper introduces DT-Fixup, a novel initialization strategy that enables training significantly deeper transformer models from scratch on small datasets, achieving state-of-the-art results on challenging Text-to-SQL and reading comprehension tasks. This challenges the long-held belief that deep transformers require massive datasets, demonstrating that optimization, not just architecture, is key to unlocking their potential.

What Problem Does This Paper Address?

The prevailing wisdom in NLP is that training deep transformer models requires vast amounts of data. Consequently, for tasks with limited labeled data, researchers typically resort to shallow neural networks or simple classifier heads on top of pre-trained models. While deeper transformers have shown promise in capturing complex reasoning and structural understanding, their training on small datasets has been largely unexplored due to perceived instability and poor generalization. This paper aims to bridge this gap by demonstrating that with proper optimization techniques, deep transformers can indeed be effectively trained on small datasets, leading to improved performance.

Methods

The core contribution of this paper is DT-Fixup, a data-dependent initialization strategy designed to stabilize the training of deep transformers, particularly in a "mixed setup" where new transformer layers are added on top of pre-trained models. Inspired by the T-Fixup method, DT-Fixup addresses limitations that prevent T-Fixup from being directly applied to this mixed scenario.

Key aspects of DT-Fixup include:

  • Data-Dependent Scaling: Instead of relying on fixed scaling factors, DT-Fixup estimates the input norm from the training data and uses this to derive more appropriate scaling for model parameters. This is crucial because the input to the new transformer layers comes from pre-trained models, whose norms are not directly controlled by the initialization.
  • Elimination of Warm-up and Layer Normalization: Similar to T-Fixup, DT-Fixup aims to remove the need for learning rate warm-up and layer normalization within the newly added transformer layers, simplifying the training process.
  • Adaptation for Relational Transformers: The method is extended to handle relation-aware transformer variants, which are particularly relevant for structured tasks like Text-to-SQL.

The paper provides theoretical analysis to justify the proposed scaling factors and demonstrates their effectiveness through empirical evaluation.

Datasets

The paper evaluates DT-Fixup on two benchmark datasets:

  • Spider: A complex and cross-domain Text-to-SQL semantic parsing benchmark. It's noted for its relatively small size (around 10,000 training samples) and the need for reasoning and structural understanding to translate natural language questions into SQL queries.
  • ReClor: A reading comprehension dataset that requires logical reasoning. Similar to Spider, it's a small dataset (around 6,000 training samples) designed to test advanced reasoning capabilities.

Key Insights and Ideas

  • Optimization, Not Just Architecture, Matters for Small Datasets: The paper strongly suggests that the perceived limitations of deep transformers on small datasets are primarily due to optimization challenges, not inherent architectural flaws. DT-Fixup demonstrates that effective optimization can unlock the benefits of depth.
  • Data-Dependent Initialization is Crucial for Mixed Setups: When adding new layers to pre-trained models, the input to these new layers is not freely initialized. DT-Fixup's data-dependent approach accounts for this, making it suitable for scenarios where T-Fixup fails.
  • Deeper Models Improve Reasoning on Hard Cases: Error analysis reveals that the performance gains from deeper models are particularly pronounced on "hard cases" that require complex reasoning and structural understanding, leading to more reasonable predictions even when errors occur.
  • Large-Batch Training Can Hurt Generalization on Small Datasets: The paper empirically verifies that large batch sizes, often used to stabilize training, can negatively impact generalization on small datasets, reinforcing the need for methods like DT-Fixup that work well with smaller batches.

Key Results

  • State-of-the-Art Performance: On the Spider Text-to-SQL benchmark, the proposed DT-Fixup SQL-SP model achieved state-of-the-art exact match accuracy of 70.9% on the test set, outperforming previous approaches without task-specific pre-training.
  • Improved Generalization: DT-Fixup consistently improved generalization across both Spider and ReClor, enabling the training of significantly deeper transformer models (up to 48 layers for Spider) that performed better than shallower models trained with standard methods.
  • Faster Convergence: Validation curves showed that models trained with DT-Fixup converged much faster than those trained with the standard optimization strategy.
  • ReClor Performance: On ReClor, adding DT-Fixup with extra transformer layers achieved near state-of-the-art performance, ranking second on the public leaderboard.
  • Error Analysis: Deeper models trained with DT-Fixup showed a significant reduction in "Both" errors (sketch and column mistakes), indicating better handling of complex cases requiring reasoning.

Limitations and Future Work

  • Dataset Size: While the paper demonstrates success on small datasets, the authors acknowledge that the datasets used (Spider and ReClor) are still small by deep learning standards. Further validation on even smaller or more diverse datasets would be beneficial.
  • Generalizability to Other Architectures: While the analysis is presented as generally applicable, the primary empirical validation is on transformer architectures. Extending the evaluation to other neural network architectures would be valuable.
  • Task-Specific Pre-training: The paper highlights that DT-Fixup achieves SOTA without task-specific pre-training. Future work could explore how DT-Fixup interacts with or complements existing task-specific pre-training strategies.
  • Further Error Analysis: The paper provides a good start with error analysis, but deeper dives into the types of reasoning failures and how DT-Fixup specifically addresses them could offer further insights.

Summary updated

BibTeX
@inproceedings{xu2021optimizing,
  author = {Peng Xu and Dhruv Kumar and Wei Yang and Wenjie Zi and Keyi Tang and Chenyang Huang and Jackie Chi Kit Cheung and Simon J. D. Prince and Yanshuai Cao},
  title = {{Optimizing Deeper Transformers on Small Datasets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=CY23CosM5xP0},
}
Figure