AutoRC: Improving BERT Based Relation Classification Models via Architecture Search

Wei Zhu, Xipeng Qiu, Yuan Ni, Guotong Xie

Abstract

Relation Classification (RC) is a fundamental task in natural language processing, yet the optimal architecture for BERT-based RC models remains elusive and often task-dependent. We present AutoRC, a framework that leverages Neural Architecture Search (NAS) to automatically design task-specific RC architectures. By exploring a search space of over 160 million combinations—including entity span identification, positional encoding, pooling strategies, and feature interactions—AutoRC identifies superior designs across diverse domains. Our results on seven benchmark datasets demonstrate that AutoRC consistently outperforms manually tuned baselines, achieving state-of-the-art performance with high efficiency.

Introduction

Relation Classification (RC) involves identifying the semantic connection between two entities within a sentence. For example, in the sentence "The kitchen is part of the house," an RC model should recognize the component-whole relationship. While BERT has become the standard backbone for RC, there is no consensus on the optimal way to extract and combine features from its hidden states. Researchers typically rely on manual trial-and-error to decide on entity markers, pooling layers, and feature interactions. AutoRC addresses this by automating the design process, recognizing that the "best" architecture varies significantly across different datasets and domains.

Methods

AutoRC utilizes a Reinforcement Learning (RL) based Neural Architecture Search to navigate a massive search space of approximately $1.64 \times 10^8$ possible BERT-based RC configurations.

The Search Space

The search space is structured into four primary stages:

  1. Entity Span Identification: Determining how to represent entity locations to the model. Options include standard input, using special markers like [E1] and [/E1], or replacing entity mentions with generic tokens like [ENTITY-1].
  2. Entity Positional Encoding: Deciding whether to incorporate additional embeddings to explicitly track the relative positions of entities.
  3. Pooling Layer: Selecting from six pooling operations (e.g., Average, Max, Self-Attention) to condense token-level representations into fixed-length vectors for entities and their context.
  4. Output Features & Interactions: Choosing which features to concatenate and whether to compute explicit interactions (e.g., absolute difference or dot product) between entity and context vectors.
The AutoRC framework architecture, illustrating the flow from BERT embeddings through pooling and interaction layers to the final softmax classification.

Search Strategy

The framework employs an LSTM-based controller that predicts a sequence of architectural choices to form a "child model." These child models are trained, and their F1 scores serve as a reward signal to update the controller's policy via the REINFORCE algorithm. To ensure computational efficiency, AutoRC uses parameter sharing, allowing candidate architectures to share underlying BERT weights.

Different strategies for entity span identification within the AutoRC search space, including standard input, entity markers, and entity replacement tokens.

Results and Discussion

The authors evaluated AutoRC across seven benchmarks, including general (SemEval-2010, Wiki80, KBP-37), biomedical (i2b2, DDI, ChemProt), and long-context (DEFT-2020) datasets.

Performance Gains

AutoRC consistently outperformed standard BERT-entity baselines. Notably, on the DEFT-2020 dataset, which features long entity spans, AutoRC improved the Macro F1 score from 60.19 to 63.58. On the i2b2 clinical dataset, it achieved a state-of-the-art F1 of 83.37, compared to the baseline's 81.94.

Key Insights

  • Task-Specific Architectures: The search revealed that different domains require different designs. For instance, replacing entity mentions with tokens was more effective for the long entities in DEFT-2020 than using markers.
  • Importance of Interactions: While many manual models rely solely on concatenation, AutoRC frequently discovered that calculating absolute differences or dot products between features provided critical signals for classification.
  • Stability through Multi-Copy BERT: To mitigate the "moving target" problem in NAS where shared parameters fluctuate, the authors maintained three separate copies of the BERT encoder during the search phase, leading to more stable training.

Conclusion

AutoRC demonstrates that automated architecture search can effectively replace manual tuning for BERT-based Relation Classification. By discovering task-specific designs that optimize feature extraction and interaction, AutoRC achieves superior performance across a wide range of domains. Future work may involve expanding the search space to include the internal parameters of the Transformer layers themselves.

Summary updated

Abstract

Although BERT based relation classification (RC) models have achieved significant improvements over the traditional deep learning models, it seems that no consensus can be reached on what is the optimal architecture. Firstly, there are multiple alternatives for entity span identification. Second, there are a collection of pooling operations to aggregate the representations of entities and contexts into fixed length vectors. Third, it is difficult to manually decide which feature vectors, including their interactions, are beneficial for classifying the relation types. In this work, we design a comprehensive search space for BERT based RC models and employ neural architecture search (NAS) method to automatically discover the design choices mentioned above. Experiments on seven benchmark RC tasks show that our method is efficient and effective in finding better architectures than the baseline BERT based RC model. Ablation study demonstrates the necessity of our search space design and the effectiveness of our search method.
BibTeX
@misc{zhu2020autorc,
  author = {Wei Zhu and Xipeng Qiu and Yuan Ni and Guotong Xie},
  title = {{AutoRC: Improving BERT Based Relation Classification Models via Architecture Search}},
  year = {2020},
  eprint = {2009.10680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.10680},
}
Figure