Abstract
Large-scale pre-trained models like BERT have achieved state-of-the-art performance in Machine Reading Comprehension (MRC) but often suffer from high computational costs and slow inference speeds. This study explores the application of Knowledge Distillation (KD) to transfer the linguistic capabilities of a BERT-base model into a more efficient QANet architecture for Chinese MRC. Our results demonstrate that the distilled QANet model achieves a significant performance boost—improving the F1 score by over 7 points—while maintaining the high inference speed and smaller parameter footprint of the traditional architecture.
Introduction
In Natural Language Processing (NLP), pre-trained transformer models such as BERT and XLNet have set new benchmarks for Machine Reading Comprehension. However, these models are often impractical for real-world production environments due to their massive parameter counts and high latency. Conversely, traditional models like QANet offer superior speed but lack the deep semantic understanding of transformers. This research aims to bridge this gap by leveraging Knowledge Distillation to create a model that combines the speed of traditional architectures with the accuracy of state-of-the-art pre-trained models.
Methodology
The authors employ a Knowledge Distillation (KD) framework involving a "Teacher" and a "Student" model.
Model Selection
- Teacher: A
BERT-basemodel fine-tuned on the target Chinese MRC task. - Student: A QANet model, selected for its efficient architecture based on convolutions and self-attention, which avoids the sequential bottlenecks of recurrent structures (LSTMs).
The Distillation Process
Rather than training the student model solely on "hard" ground-truth labels, the student is trained to mimic the teacher's full probability distribution over output spans. By minimizing the Kullback-Leibler (KL) Divergence between the teacher's and student's outputs, the student captures "dark knowledge"—the relative uncertainties and relationships between different candidate answers—that is absent in standard supervised training.
Handling Long Documents
To accommodate BERT's 512-token input limit, a preprocessing pipeline was implemented for long Chinese documents. Text is segmented into sentences, and TF-IDF is used to calculate similarity between the question and each segment. The top 10 highest-scoring sentences are selected to form a condensed context, which successfully captured the answer span in 83.3% of cases.
Datasets
Experiments were conducted on the CMRC2018 dataset, a benchmark for Chinese span-extraction MRC consisting of human-annotated Wikipedia paragraphs. The dataset includes a training set of 10,321 questions and a validation set of 3,351 questions.
Key Results
The application of distillation significantly enhanced the performance of the traditional QANet model on the CMRC2018 validation set.
As shown in the results, distillation improved the QANet's F1 score by approximately 7.3 points and its Exact Match (EM) score by 6.7 points. Beyond accuracy, the distilled model offers substantial practical advantages:
- Efficiency: The distilled QANet is nearly 4x smaller than the BERT teacher (~30M vs ~110M parameters).
- Speed: Inference is roughly 42% faster than BERT on identical hardware, making it a viable candidate for low-latency applications.
Conclusion and Future Work
While Knowledge Distillation significantly narrows the performance gap between traditional models and transformers, a 5.5-point F1 gap remains between the distilled QANet and the BERT teacher. Future research will focus on more sophisticated distillation techniques to further close this gap. Additionally, improvements to the retrieval stage are needed, as the current TF-IDF method fails to capture the answer in approximately 16.7% of long-document cases.
Summary updated