Introduction
Grammatical Error Detection (GED) is a critical natural language processing task focused on identifying incorrect tokens within a sentence. While the advent of pre-trained Transformer models like BERT has significantly advanced the field, standard fine-tuning practices typically rely solely on the representations from the final layer. This approach overlooks the rich, hierarchical linguistic information captured across the network's depth. This paper introduces Multi-Head Multi-Layer Attention (MHMLA), a novel mechanism designed to selectively aggregate information from all layers of a pre-trained model to optimize GED performance.
Methodology
The core limitation of using only the final layer is the assumption that the highest level of abstraction is always the most relevant. Linguistic theory suggests that lower layers capture local syntax (e.g., morphology), while higher layers capture broader semantics. Since grammatical errors vary from simple spelling mistakes to complex structural issues, a model benefits from accessing multiple levels of representation simultaneously.
MHMLA treats the outputs of all $L$ layers as a sequence of representations. For every token, the model calculates an attention score across these layers, allowing it to dynamically weight the most relevant information for a specific context. By employing a multi-head mechanism, the model can simultaneously attend to different combinations of layers—for instance, one head might focus on syntactic cues in lower layers while another focuses on semantic context in upper layers. Both MHMLA and the underlying BERT parameters are fine-tuned jointly on the GED task.
Experimental Setup
The model was evaluated across three major English learner corpora:
- FCE (First Certificate in English): Short essays by language learners.
- CoNLL14: Technical essays manually corrected by two separate annotators (reported as CoNLL14-1 and CoNLL14-2).
- JFLEG (JHU FLuency-Extended GUG): A corpus emphasizing fluency and native-sounding corrections.
Results and Discussion
MHMLA achieved new state-of-the-art (SOTA) benchmarks across all tested datasets, measured primarily by the $F_{0.5}$ score to prioritize precision over recall.
As shown in the results table, MHMLA consistently outperformed previous SOTA models, with gains of 6.0 points on FCE, 8.2 points on CoNLL14-2, and 12.2 points on JFLEG. Notably, MHMLA also provided a consistent boost of 1.3 to 2.2 $F_{0.5}$ points over standard BERT-Base fine-tuning.
Key findings include:
- Dynamic Information Retrieval: The attention mechanism allows the model to adjust its focus per token, rather than using static layer weights.
- Head Count Impact: Increasing the number of attention heads generally improved performance. With more heads (8-12), the model learned to distribute its attention more broadly across the network's depth, whereas fewer heads tended to focus on specific, isolated layers.
- The "Final Layer" Trap: Even a simple average of all layers (AvgL) often outperformed the final-layer-only baseline, confirming that valuable information for GED is distributed throughout the model.
Conclusion
By moving beyond the final layer of pre-trained models, MHMLA demonstrates that multi-layer attention is a powerful tool for tasks requiring diverse linguistic features. While this study focused on BERT-Base, the mechanism is architecturally flexible and could potentially enhance other deep language representations in future research.
Summary updated