Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models
Lists
Only you see this
Abstract
In this work, we propose a method that combines two popular research areas by injecting linguistic structures into pre-trained language models in the parameter-efficient fine-tuning (PEFT) setting. In our approach, parallel adapter modules encoding different linguistic structures are combined using a novel Mixture-of-Linguistic-Experts architecture, where Gumbel-Softmax gates are used to determine the importance of these modules at each layer of the model. To reduce the number of parameters, we first train the model for a fixed small number of steps before pruning the experts based on their importance scores. Our experiment results with three different pre-trained models show that our approach can outperform state-of-the-art PEFT methods with a comparable number of parameters. In addition, we provide additional analysis to examine the experts selected by each model at each layer to provide insights for future studies.
BibTeX
@inproceedings{li2023mixture,
author = {Raymond Li and Gabriel Murray and Giuseppe Carenini},
title = {{Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models}},
booktitle = {Conference on Empirical Methods in Natural Language Processing},
year = {2023},
url = {https://openreview.net/forum?id=Q9BLbN1p6h},
}