AutoRC: Improving BERT Based Relation Classification Models via Architecture Search

Wei Zhu

Abstract

Wei Zhu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop.

2021.

Relation Classification (RC)—identifying the semantic relationship between two entities in a sentence—has been significantly enhanced by pre-trained models like BERT. However, the optimal architectural configuration for downstream RC tasks remains an open question. This paper introduces AutoRC, a customized Neural Architecture Search (NAS) framework designed to automatically discover the most effective BERT-based architectures. By exploring a search space of over 160 million configurations, AutoRC consistently outperforms manually designed state-of-the-art baselines across eight diverse benchmarks.

Introduction

Identifying semantic relationships between entities is a core task in Natural Language Processing. While BERT provides a powerful foundation, building an effective RC model requires making numerous design decisions: how to represent entities in the input, which pooling strategies to use for feature aggregation, and how to model interactions between entity vectors. The resulting search space of approximately $1.64 \times 10^8$ combinations makes manual tuning impractical. AutoRC addresses this by automating the architecture design process using reinforcement learning.

Methodology

AutoRC utilizes a modified version of Efficient Neural Architecture Search (ENAS). A recurrent neural network (RNN) acts as a "controller," sampling candidate architectures which are then trained and evaluated. The controller's policy is updated via reinforcement learning based on the performance (reward) of these child models.

Overview of the AutoRC model architecture, highlighting the modular components such as poolers and interaction layers that are subject to search.

To adapt NAS for BERT-based tasks, three key innovations were introduced:

  1. Heterogeneous Parameter Sharing: Unlike standard ENAS, weights are only shared between modules serving similar functional roles, preventing interference between disparate components.
  2. Multiple Shared Parameter Copies: To mitigate overfitting to specific architectures during the search phase, the system maintains and randomly draws from multiple copies of shared weights (e.g., the BERT encoder).
  3. Search Warm-ups: Child models undergo initial training epochs before the controller begins policy updates, ensuring more stable reward signals.

The Search Space

The framework explores various dimensions of model design, including input representations, pooling methods (max, average, or self-attention), and interaction types (e.g., dot product or absolute difference).

Comparison of input representation strategies: (a) standard input, (b) entity markers, and (c) entity tokens.

A critical finding was that the "best" representation is task-dependent. For instance, in datasets with long entity mentions like DEFT-2020, replacing entities with special tokens (Figure 3c) often outperformed using markers (Figure 3b), as it allowed the model to focus on structural context rather than specific entity names.

Experimental Results

AutoRC was evaluated on eight benchmarks across general, scientific, and medical domains (e.g., SemEval-2010, TACRED, ChemProt).

  • Performance: AutoRC achieved superior F1 scores compared to strong baselines like R-BERT and BERT-entity on all datasets.
  • Interaction Importance: The search process revealed that including interactions like the dot product between entity and context vectors provides a significant performance boost, a feature often overlooked in manual designs.
  • Transferability: Beyond RC, the framework showed strong results in Coreference Resolution, outperforming SpanBERT on the OntoNotes benchmark (86.1 vs 85.3 MUC F1).

Conclusion and Limitations

AutoRC demonstrates that automated architecture search can significantly improve the performance and robustness of BERT-based models for Relation Classification. While the search process is computationally intensive—requiring up to two NVIDIA V100 GPUs per task—the discovery of task-specific optimal components suggests that a "one-size-fits-all" approach to RC is suboptimal. Future work will focus on expanding the search space beyond human-chosen modules and improving the efficiency of the NAS process.

Summary updated

Abstract

Wei Zhu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. 2021.
BibTeX
@misc{zhu2020autorc,
  author = {Wei Zhu},
  title = {{AutoRC: Improving BERT Based Relation Classification Models via Architecture Search}},
  year = {2020},
  eprint = {2009.10680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.10680},
}
Figure