TL;DR
This paper introduces the Hidden Representation Extractor (HIRE), a novel component that enhances pre-trained language models by adaptively leveraging information from all hidden layers, not just the final one. By fusing these deeper representations with the final layer's output, HIRE significantly improves performance on various natural language understanding tasks, rivaling state-of-the-art models.
What Problem Does This Paper Address?
Transformer-based language models like BERT and RoBERTa have revolutionized NLP by learning rich contextual representations. However, a common practice during fine-tuning for downstream tasks is to exclusively use the output of the final encoder layer. The authors argue that this approach is suboptimal, as valuable linguistic information might be encoded in the intermediate layers. They hypothesize that these earlier layers capture different aspects of language, and that solely relying on the last layer might limit the model's full potential. This paper aims to unlock the latent power within these intermediate representations.
Methods
The core innovation is the Hidden Representation Extractor (HIRE), a module designed to capture "complementary information" from the Transformer's hidden layers that the final layer might miss. HIRE works in two main steps:
- Adaptive Importance Scoring: For each hidden layer's output, HIRE calculates an "importance score." This score is learned dynamically using a small network (a 2-layer Bidirectional GRU followed by a linear layer and ReLU activation) that takes a summarized representation of each hidden state. This allows the model to weigh the contribution of each layer differently based on the task and even the specific input.
- Complementary Representation Generation: The importance scores are normalized using softmax to create a probability distribution. This distribution is then used to compute a weighted sum of all hidden layer representations, resulting in a "complementary representation" (denoted as
A).
This complementary representation A is then fused with the original model's final layer output R using a Fusion Network. This fusion involves concatenation, element-wise addition, and element-wise multiplication of R and A. The fused representation is then passed through a 2-layer Bidirectional GRU to further integrate the information before being fed to a task-specific output layer. The authors use RoBERTa-large as their backbone model.
Datasets
The experiments were conducted on the GLUE benchmark, a collection of nine diverse datasets for evaluating natural language understanding models. These datasets cover tasks such as:
- Single-sentence tasks: Corpus of Linguistic Acceptability (CoLA), Stanford Sentiment Treebank (SST-2).
- Similarity and paraphrase tasks: Microsoft Research Paraphrase Corpus (MRPC), Quora Question Pairs (QQP), Semantic Textual Similarity Benchmark (STS-B).
- Natural Language Inference (NLI) tasks: Multi-Genre Natural Language Inference Corpus (MNLI), Stanford Question Answering Dataset (QNLI), Recognizing Textual Entailment (RTE), and Winograd Schema Challenge (WNLI).
The paper reports results on both the development and test sets of these benchmarks.
Key Insights and Ideas
- Intermediate layers are valuable: The paper strongly supports the idea that intermediate layers of large Transformer models encode crucial linguistic information that is not fully captured by the final layer alone.
- Dynamic weighting is key: Simply averaging or using fixed combinations of layer outputs is less effective than dynamically learning which layers are most important for a given task and input. HIRE's adaptive mechanism is a significant contributor to its success.
- Fusion is more than concatenation: The fusion network's use of concatenation, addition, and Hadamard product, followed by a GRU, effectively integrates the original and complementary representations.
- Task-specific adaptation: The importance scores learned by HIRE vary across different NLU tasks, demonstrating its ability to adapt to diverse linguistic challenges.
Key Results
- The proposed RoBERTa+HIRE model consistently outperforms the baseline RoBERTa model on several GLUE development set tasks, with notable improvements on CoLA (+1.7 points) and SST-2 (+0.4 points).
- On the GLUE test set, RoBERTa+HIRE shows improvements across multiple tasks, including CoLA (+0.8 points), SST-2 (+0.4 points), and MRPC (+0.7/+0.9 points).
- The model achieves state-of-the-art results on the SST-2 dataset with a score of 97.1.
- Ablation studies confirm that both HIRE and the fusion network are crucial for performance gains. Removing HIRE leads to a significant drop in performance (e.g., 1.2 points on CoLA), while removing the fusion network also causes a noticeable decrease (e.g., 1.5 points on CoLA).
- The dynamic importance scoring mechanism in HIRE is shown to be superior to fixed strategies (e.g., averaging all layers, using only the first/last 6 layers).
Limitations and Future Work
- WNLI dataset issues: The paper notes that the WNLI dataset is problematic and its results are excluded from the main tables.
- Parameter increase: While HIRE is described as a "lite" component, it does add parameters to the model (e.g., an increase from 355M to 437M parameters for RoBERTa+HIRE compared to RoBERTa). The authors do not explicitly discuss the trade-off between performance gains and parameter count in detail, but the parameter increase is relatively small compared to the overall model size.
- Future work: The authors suggest that their approach of adaptively leveraging hidden representations could be applied to other Transformer-based models and potentially other architectures. The analysis of importance scores across different tasks and examples hints at deeper insights into how these models learn and represent language, which could be further explored.
Summary updated