Abstract
Relation Classification (RC) is a fundamental task in natural language processing, yet the optimal architecture for BERT-based RC models remains elusive and often task-dependent. We present AutoRC, a framework that leverages Neural Architecture Search (NAS) to automatically design task-specific RC architectures. By exploring a search space of over 160 million combinations—including entity span identification, positional encoding, pooling strategies, and feature interactions—AutoRC identifies superior designs across diverse domains. Our results on seven benchmark datasets demonstrate that AutoRC consistently outperforms manually tuned baselines, achieving state-of-the-art performance with high efficiency.
Introduction
Relation Classification (RC) involves identifying the semantic connection between two entities within a sentence. For example, in the sentence "The kitchen is part of the house," an RC model should recognize the component-whole relationship. While BERT has become the standard backbone for RC, there is no consensus on the optimal way to extract and combine features from its hidden states. Researchers typically rely on manual trial-and-error to decide on entity markers, pooling layers, and feature interactions. AutoRC addresses this by automating the design process, recognizing that the "best" architecture varies significantly across different datasets and domains.
Methods
AutoRC utilizes a Reinforcement Learning (RL) based Neural Architecture Search to navigate a massive search space of approximately $1.64 \times 10^8$ possible BERT-based RC configurations.
The Search Space
The search space is structured into four primary stages:
- Entity Span Identification: Determining how to represent entity locations to the model. Options include standard input, using special markers like
[E1]and[/E1], or replacing entity mentions with generic tokens like[ENTITY-1]. - Entity Positional Encoding: Deciding whether to incorporate additional embeddings to explicitly track the relative positions of entities.
- Pooling Layer: Selecting from six pooling operations (e.g., Average, Max, Self-Attention) to condense token-level representations into fixed-length vectors for entities and their context.
- Output Features & Interactions: Choosing which features to concatenate and whether to compute explicit interactions (e.g., absolute difference or dot product) between entity and context vectors.
Search Strategy
The framework employs an LSTM-based controller that predicts a sequence of architectural choices to form a "child model." These child models are trained, and their F1 scores serve as a reward signal to update the controller's policy via the REINFORCE algorithm. To ensure computational efficiency, AutoRC uses parameter sharing, allowing candidate architectures to share underlying BERT weights.
Results and Discussion
The authors evaluated AutoRC across seven benchmarks, including general (SemEval-2010, Wiki80, KBP-37), biomedical (i2b2, DDI, ChemProt), and long-context (DEFT-2020) datasets.
Performance Gains
AutoRC consistently outperformed standard BERT-entity baselines. Notably, on the DEFT-2020 dataset, which features long entity spans, AutoRC improved the Macro F1 score from 60.19 to 63.58. On the i2b2 clinical dataset, it achieved a state-of-the-art F1 of 83.37, compared to the baseline's 81.94.
Key Insights
- Task-Specific Architectures: The search revealed that different domains require different designs. For instance, replacing entity mentions with tokens was more effective for the long entities in DEFT-2020 than using markers.
- Importance of Interactions: While many manual models rely solely on concatenation, AutoRC frequently discovered that calculating absolute differences or dot products between features provided critical signals for classification.
- Stability through Multi-Copy BERT: To mitigate the "moving target" problem in NAS where shared parameters fluctuate, the authors maintained three separate copies of the BERT encoder during the search phase, leading to more stable training.
Conclusion
AutoRC demonstrates that automated architecture search can effectively replace manual tuning for BERT-based Relation Classification. By discovering task-specific designs that optimize feature extraction and interaction, AutoRC achieves superior performance across a wide range of domains. Future work may involve expanding the search space to include the internal parameters of the Transformer layers themselves.
Summary updated