Abstract
Wei Zhu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop.
2021.
Relation Classification (RC)—identifying the semantic relationship between two entities in a sentence—has been significantly enhanced by pre-trained models like BERT. However, the optimal architectural configuration for downstream RC tasks remains an open question. This paper introduces AutoRC, a customized Neural Architecture Search (NAS) framework designed to automatically discover the most effective BERT-based architectures. By exploring a search space of over 160 million configurations, AutoRC consistently outperforms manually designed state-of-the-art baselines across eight diverse benchmarks.
Introduction
Identifying semantic relationships between entities is a core task in Natural Language Processing. While BERT provides a powerful foundation, building an effective RC model requires making numerous design decisions: how to represent entities in the input, which pooling strategies to use for feature aggregation, and how to model interactions between entity vectors. The resulting search space of approximately $1.64 \times 10^8$ combinations makes manual tuning impractical. AutoRC addresses this by automating the architecture design process using reinforcement learning.
Methodology
AutoRC utilizes a modified version of Efficient Neural Architecture Search (ENAS). A recurrent neural network (RNN) acts as a "controller," sampling candidate architectures which are then trained and evaluated. The controller's policy is updated via reinforcement learning based on the performance (reward) of these child models.
To adapt NAS for BERT-based tasks, three key innovations were introduced:
- Heterogeneous Parameter Sharing: Unlike standard ENAS, weights are only shared between modules serving similar functional roles, preventing interference between disparate components.
- Multiple Shared Parameter Copies: To mitigate overfitting to specific architectures during the search phase, the system maintains and randomly draws from multiple copies of shared weights (e.g., the BERT encoder).
- Search Warm-ups: Child models undergo initial training epochs before the controller begins policy updates, ensuring more stable reward signals.
The Search Space
The framework explores various dimensions of model design, including input representations, pooling methods (max, average, or self-attention), and interaction types (e.g., dot product or absolute difference).
A critical finding was that the "best" representation is task-dependent. For instance, in datasets with long entity mentions like DEFT-2020, replacing entities with special tokens (Figure 3c) often outperformed using markers (Figure 3b), as it allowed the model to focus on structural context rather than specific entity names.
Experimental Results
AutoRC was evaluated on eight benchmarks across general, scientific, and medical domains (e.g., SemEval-2010, TACRED, ChemProt).
- Performance: AutoRC achieved superior F1 scores compared to strong baselines like R-BERT and BERT-entity on all datasets.
- Interaction Importance: The search process revealed that including interactions like the dot product between entity and context vectors provides a significant performance boost, a feature often overlooked in manual designs.
- Transferability: Beyond RC, the framework showed strong results in Coreference Resolution, outperforming SpanBERT on the OntoNotes benchmark (86.1 vs 85.3 MUC F1).
Conclusion and Limitations
AutoRC demonstrates that automated architecture search can significantly improve the performance and robustness of BERT-based models for Relation Classification. While the search process is computationally intensive—requiring up to two NVIDIA V100 GPUs per task—the discovery of task-specific optimal components suggests that a "one-size-fits-all" approach to RC is suboptimal. Future work will focus on expanding the search space beyond human-chosen modules and improving the efficiency of the NAS process.
Summary updated