Optimizing Deeper Transformers on Small Datasets: An Application on Text-to-SQL Semantic Parsing

Peng Xu, Wei Yang, Wenjie Zi, Keyi Tang, Chengyang Huang, Jackie Chi Kit Cheung, Yanshuai Cao

Abstract

It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train $48$ layers of transformers, comprising $24$ fine-tuned layers from pre-trained RoBERTa and $24$ relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider. We achieve this by deriving a novel Data-dependent Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the prior T-Fixup work. Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.

Introduction: The Challenge of Depth on Small Data

In Natural Language Processing (NLP), the prevailing wisdom suggests that "deeper is better" only when massive datasets are available. When working with small datasets (typically under 10,000 samples), researchers often resort to shallow "heads" on top of pre-trained models. Attempting to train deep Transformer layers from scratch in these low-resource settings frequently results in unstable optimization, poor generalization, or a total failure to converge.

This paper argues that this is not an inherent limitation of the Transformer architecture itself, but rather a consequence of standard optimization techniques. Traditional approaches rely on a fragile balance of Layer Normalization, learning rate warm-up, and large batch sizes. On small datasets, large batches can degrade generalization, and the limited computational resources often available for such tasks make these components difficult to tune effectively.

Methodology: DT-Fixup

The authors introduce DT-Fixup (Data-dependent Transformer Fixed-update initialization) to overcome these hurdles. Building upon the T-Fixup technique, DT-Fixup addresses two critical gaps:

  1. Data Dependency: Unlike T-Fixup, which assumes Gaussian inputs, DT-Fixup recognizes that when stacking new layers on a pre-trained model, the input norm is fixed by that model. DT-Fixup uses a single forward pass over the training data to estimate the maximum input norm ($\mu$), which is then used to scale initial weights.
  2. Relational Encodings: The framework is extended to support Relation-Aware Transformers (RAT), which are essential for Text-to-SQL tasks where understanding structural links between questions and database schemas is paramount.

By employing DT-Fixup, the authors can eliminate Layer Normalization and learning rate warm-up in the newly added layers. This stabilizes gradient updates, allowing for faster and more reliable convergence even with small batch sizes.

Figure 2: Comparison of training stability and convergence speed between DT-Fixup and standard optimization across different layer depths.

Experimental Setup

The approach was evaluated on two rigorous benchmarks:

  • Spider: A complex, cross-domain Text-to-SQL semantic parsing dataset containing approximately 10,000 questions across 200 databases.
  • ReClor: A reading comprehension dataset requiring logical reasoning, consisting of 6,000 questions from standardized exams like the GMAT and LSAT.

Key Insights and Results

The primary finding is that the failure of deep models on small datasets is an optimization problem, not an architectural one. With proper initialization, the benefits of depth—specifically improved reasoning and structural understanding—are accessible even when data is scarce.

Figure 5: Performance comparison on the Spider dataset. DT-Fixup enables training much deeper models (24 layers) without task-specific pre-training, achieving state-of-the-art results.

Performance Highlights

  • State-of-the-Art Results: The model achieved 70.9% exact match accuracy on the Spider test set, a top-tier performance at the time of publication.
  • Efficiency and Depth: The authors successfully trained a model with 24 relation-aware layers on top of 24 RoBERTa layers (48 total). In contrast, standard optimization failed to train models deeper than 8 layers on the same data.
  • No Task-Specific Pre-training: Unlike competing models (e.g., GraPPa or GAP) that require massive task-specific pre-training, this approach reached SOTA levels using only standard RoBERTa and DT-Fixup.
  • Reasoning on "Hard" Cases: Error analysis revealed that the 48-layer model's gains were most significant in "Hard" and "Extra Hard" cases, suggesting that depth is key to handling implicit reasoning in complex queries.

Conclusion and Future Work

This work demonstrates that deep Transformers can be effectively trained on small datasets by focusing on initialization and optimization. While the current implementation focuses on the encoder and uses a standard LSTM-based decoder, extending DT-Fixup to deep Transformer decoders is a promising future direction. Additionally, while the data-dependent pass to calculate $\mu$ is computationally inexpensive, future research could explore dynamic estimation methods to make the initialization purely architectural.

Summary updated

Abstract

It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train $48$ layers of transformers, comprising $24$ fine-tuned layers from pre-trained RoBERTa and $24$ relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider. We achieve this by deriving a novel Data-dependent Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the prior T-Fixup work. Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.
BibTeX
@inproceedings{xu2021optimizing,
  author = {Peng Xu and Wei Yang and Wenjie Zi and Keyi Tang and Chengyang Huang and Jackie Chi Kit Cheung and Yanshuai Cao},
  title = {{Optimizing Deeper Transformers on Small Datasets: An Application on Text-to-SQL Semantic Parsing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=CY23CosM5xP0},
}
Figure