Multi-Head Multi-Layer Attention to Deep Language Representations for Grammatical Error Detection

Masahiro Kaneko, Mamoru Komachi

Introduction

Grammatical Error Detection (GED) is a critical natural language processing task focused on identifying incorrect tokens within a sentence. While the advent of pre-trained Transformer models like BERT has significantly advanced the field, standard fine-tuning practices typically rely solely on the representations from the final layer. This approach overlooks the rich, hierarchical linguistic information captured across the network's depth. This paper introduces Multi-Head Multi-Layer Attention (MHMLA), a novel mechanism designed to selectively aggregate information from all layers of a pre-trained model to optimize GED performance.

Methodology

The core limitation of using only the final layer is the assumption that the highest level of abstraction is always the most relevant. Linguistic theory suggests that lower layers capture local syntax (e.g., morphology), while higher layers capture broader semantics. Since grammatical errors vary from simple spelling mistakes to complex structural issues, a model benefits from accessing multiple levels of representation simultaneously.

Comparison of standard BERT fine-tuning (a) and the proposed MHMLA architecture (b).

MHMLA treats the outputs of all $L$ layers as a sequence of representations. For every token, the model calculates an attention score across these layers, allowing it to dynamically weight the most relevant information for a specific context. By employing a multi-head mechanism, the model can simultaneously attend to different combinations of layers—for instance, one head might focus on syntactic cues in lower layers while another focuses on semantic context in upper layers. Both MHMLA and the underlying BERT parameters are fine-tuned jointly on the GED task.

Experimental Setup

The model was evaluated across three major English learner corpora:

  • FCE (First Certificate in English): Short essays by language learners.
  • CoNLL14: Technical essays manually corrected by two separate annotators (reported as CoNLL14-1 and CoNLL14-2).
  • JFLEG (JHU FLuency-Extended GUG): A corpus emphasizing fluency and native-sounding corrections.

Results and Discussion

MHMLA achieved new state-of-the-art (SOTA) benchmarks across all tested datasets, measured primarily by the $F_{0.5}$ score to prioritize precision over recall.

Performance comparison of MHMLA against previous SOTA methods and standard BERT baselines.

As shown in the results table, MHMLA consistently outperformed previous SOTA models, with gains of 6.0 points on FCE, 8.2 points on CoNLL14-2, and 12.2 points on JFLEG. Notably, MHMLA also provided a consistent boost of 1.3 to 2.2 $F_{0.5}$ points over standard BERT-Base fine-tuning.

Key findings include:

  • Dynamic Information Retrieval: The attention mechanism allows the model to adjust its focus per token, rather than using static layer weights.
  • Head Count Impact: Increasing the number of attention heads generally improved performance. With more heads (8-12), the model learned to distribute its attention more broadly across the network's depth, whereas fewer heads tended to focus on specific, isolated layers.
  • The "Final Layer" Trap: Even a simple average of all layers (AvgL) often outperformed the final-layer-only baseline, confirming that valuable information for GED is distributed throughout the model.

Conclusion

By moving beyond the final layer of pre-trained models, MHMLA demonstrates that multi-layer attention is a powerful tool for tasks requiring diverse linguistic features. While this study focused on BERT-Base, the mechanism is architecturally flexible and could potentially enhance other deep language representations in future research.

Summary updated

Abstract

It is known that a deep neural networkmodel pre-trained with large-scale data greatly improves the accuracy of various tasks, especially when there are resource constraints. However, the information needed to solve a given task can vary, and simplyusing the output of the final layer is not necessarily sufficient. Moreover, to our knowledge, exploiting large language representation models to detect grammatical errors has not yet been studied. In this work, we investigate the effect of utilizing information not only from the final layer but also from intermediate layers ofa pre-trained language representation model to detect grammatical errors. We propose a multi-head multi-layer attention model that determines the appropriate layers in Bidirectional Encoder Representation from Transformers (BERT). The proposed method achieved the best scoreson three datasets for grammatical error detection tasks, outperforming the current state of the art method by 6.0 points on FCE, 8.2 points on CoNLL14, and 12.2 pointson JFLEG in terms of F0.5. We also demonstrate that by using multi-head multi-layer attention, our model can exploit a broader range of information for each token in a sentence than a model that uses only the final layer’s information.
BibTeX
@article{kaneko2019multi,
  author = {Masahiro Kaneko and Mamoru Komachi},
  title = {{Multi-Head Multi-Layer Attention to Deep Language Representations for Grammatical Error Detection}},
  journal = {Computación y Sistemas},
  year = {2019},
  doi = {10.13053/cys-23-3-3271},
}
Figure