Introduction
In medical question-answering (QA) systems, Natural Language Inference (NLI) serves as a critical component for matching user queries to known intents or FAQs. However, the optimal model architecture often depends heavily on the specific dataset. Manual tuning is time-consuming, and while large-scale models like BERT offer high accuracy, their computational demands often preclude real-time deployment. To address these challenges, researchers have developed NASQU (Neural Architecture Search for Query Understanding), an automated framework designed to discover high-performing, production-ready architectures for medical NLI.
The NASQU Framework
NASQU is built upon the Efficient Neural Architecture Search (ENAS) framework but introduces several domain-specific enhancements:
- Expanded Search Space: Unlike standard NLU search spaces, NASQU incorporates cross-sentence attention modules. This allows the model to directly align the semantics of sentence pairs (premise and hypothesis), which is vital for NLI tasks.
- Aggregator Search: Instead of relying on fixed pooling methods, NASQU searches for optimal ways to aggregate encoder outputs, selecting from options like self-attention, dynamic routing, or standard pooling.
- Efficiency-Aware Search: To ensure production viability, the search process penalizes models that exceed 1GB of GPU memory or 20ms of inference time.
- Knowledge Distillation: A pre-trained Chinese BERT model acts as a "teacher," guiding the smaller searched architectures during both the search and final training phases to maximize accuracy.
Key Insights: Task-Specific Architectures
The search process revealed that "one size fits all" is sub-optimal for medical NLI. The framework discovered two distinct models tailored to different data characteristics: NASQU-1 and NASQU-2.
NASQU-1, optimized for the larger CMFAQ dataset, utilizes a more complex structure with multi-head attention and convolutional layers. In contrast, NASQU-2 is more lightweight, designed for the smaller CMQP dataset to prevent overfitting. A common theme across both models is the use of hybrid layers—combining CNNs for local features with GRUs or Attention for long-range dependencies—and the consistent selection of cross-sentence attention modules.
Performance and Results
The automated search yielded models that significantly outperform traditional baselines like ESIM and rival the performance of much larger models while maintaining production-level efficiency.
As shown in the results, NASQU-1 achieved an 87.1 F1 score on the CMFAQ dataset. While slightly behind the BERT-wwm-ext teacher model, the NASQU architectures offer a superior balance for production environments:
- Latency: Inference takes only 11–14ms per sample, compared to 66ms for BERT.
- Memory Footprint: NASQU models require approximately 0.7–0.8 GB of memory, whereas BERT requires 4.8 GB.
Conclusion
NASQU demonstrates that automated architecture search can effectively bridge the gap between high-accuracy research models and high-efficiency production requirements. By incorporating cross-sentence attention and aggregator search, the framework discovers task-specific motifs that outperform manual designs. Future work will explore the generalizability of these medical-specific motifs to other languages and broader NLP domains.
Summary updated