Discovering Better Model Architectures for Medical Query Understanding

Wei Zhu, Yuan Ni, Xiaoling Wang, Guotong Xie

Introduction

In medical question-answering (QA) systems, Natural Language Inference (NLI) serves as a critical component for matching user queries to known intents or FAQs. However, the optimal model architecture often depends heavily on the specific dataset. Manual tuning is time-consuming, and while large-scale models like BERT offer high accuracy, their computational demands often preclude real-time deployment. To address these challenges, researchers have developed NASQU (Neural Architecture Search for Query Understanding), an automated framework designed to discover high-performing, production-ready architectures for medical NLI.

The NASQU Framework

NASQU is built upon the Efficient Neural Architecture Search (ENAS) framework but introduces several domain-specific enhancements:

  • Expanded Search Space: Unlike standard NLU search spaces, NASQU incorporates cross-sentence attention modules. This allows the model to directly align the semantics of sentence pairs (premise and hypothesis), which is vital for NLI tasks.
  • Aggregator Search: Instead of relying on fixed pooling methods, NASQU searches for optimal ways to aggregate encoder outputs, selecting from options like self-attention, dynamic routing, or standard pooling.
  • Efficiency-Aware Search: To ensure production viability, the search process penalizes models that exceed 1GB of GPU memory or 20ms of inference time.
  • Knowledge Distillation: A pre-trained Chinese BERT model acts as a "teacher," guiding the smaller searched architectures during both the search and final training phases to maximize accuracy.

Key Insights: Task-Specific Architectures

The search process revealed that "one size fits all" is sub-optimal for medical NLI. The framework discovered two distinct models tailored to different data characteristics: NASQU-1 and NASQU-2.

Discovered architectures for NASQU-1 and NASQU-2, showing the combination of convolutional, recurrent, and attention layers.

NASQU-1, optimized for the larger CMFAQ dataset, utilizes a more complex structure with multi-head attention and convolutional layers. In contrast, NASQU-2 is more lightweight, designed for the smaller CMQP dataset to prevent overfitting. A common theme across both models is the use of hybrid layers—combining CNNs for local features with GRUs or Attention for long-range dependencies—and the consistent selection of cross-sentence attention modules.

Performance and Results

The automated search yielded models that significantly outperform traditional baselines like ESIM and rival the performance of much larger models while maintaining production-level efficiency.

Comparison of F1 scores between NASQU models and the BERT teacher model across two medical datasets.

As shown in the results, NASQU-1 achieved an 87.1 F1 score on the CMFAQ dataset. While slightly behind the BERT-wwm-ext teacher model, the NASQU architectures offer a superior balance for production environments:

  • Latency: Inference takes only 11–14ms per sample, compared to 66ms for BERT.
  • Memory Footprint: NASQU models require approximately 0.7–0.8 GB of memory, whereas BERT requires 4.8 GB.

Conclusion

NASQU demonstrates that automated architecture search can effectively bridge the gap between high-accuracy research models and high-efficiency production requirements. By incorporating cross-sentence attention and aggregator search, the framework discovers task-specific motifs that outperform manual designs. Future work will explore the generalizability of these medical-specific motifs to other languages and broader NLP domains.

Summary updated

Abstract

In developing an online question-answering system for the medical domains, natural language inference (NLI) models play a central role in question matching and intention detection. However, which models are best for our datasets? Manually selecting or tuning a model is time-consuming. Thus we experiment with automatically optimizing the model architectures on the task at hand via neural architecture search (NAS). First, we formulate a novel architecture search space based on the previous NAS literature, supporting cross-sentence attention (cross-attn) modeling. Second, we propose to modify the ENAS method to accelerate and stabilize the search results. We conduct extensive experiments on our two medical NLI tasks. Results show that our system can easily outperform the classical baseline models. We compare different NAS methods and demonstrate our approach provides the best results.
BibTeX
@inproceedings{zhu2021discovering,
  author = {Wei Zhu and Yuan Ni and Xiaoling Wang and Guotong Xie},
  title = {{Discovering Better Model Architectures for Medical Query Understanding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-industry.29},
}
Figure