Abstract
Machine Reading Comprehension (MRC) of abstract concepts presents a unique challenge due to the difficulty of grounding non-perceivable terms and the necessity of long-range context. This paper describes the NLP-IIS@UT system for SemEval-2021 Task 4, which utilizes the Longformer architecture to expand the model's receptive field. By employing linear scaling attention and global attention mechanisms, the system achieves significant performance improvements over standard BERT baselines, reaching over 70% accuracy on abstract reasoning tasks.
Introduction
Traditional MRC tasks often focus on concrete entities and events. However, the Reading Comprehension of Abstract Meaning (ReCAM) task requires models to understand concepts that are either imperceivable (e.g., culture) or highly non-specific (e.g., entity). Two primary hurdles exist: the inherent difficulty in representing abstract words and the "receptive field bottleneck" of standard transformers like BERT, which are limited to 512 tokens. If a model cannot process the entire document at once, it misses the fine-grained connections needed to grasp abstract meaning.
The ReCAM Task
The ReCAM task evaluates a model's ability to fill in a placeholder within a summary or question using an abstract word derived from a long passage.
The dataset is divided into two subtasks:
- Subtask 1 (Imperceptibility): Focuses on ideas distant from immediate physical perception (e.g., politics, values).
- Subtask 2 (Non-specificity): Focuses on high-level taxonomic categories (e.g., using animal instead of dog).
Methodology
To address the limitations of standard transformers, the authors employed the Longformer (Long Document Transformer), drawing on techniques proven effective for long-document processing on benchmarks like WikiHop.
Long-Range Attention
Unlike standard Transformers with quadratic complexity, Longformer uses a specialized attention mechanism that scales linearly, allowing for a context window of up to 4,096 tokens. This is crucial for ReCAM, where abstract meaning is often distributed across an entire document.
Model Architecture
The system incorporates an attentional transformation layer on top of the base model to better integrate the question and candidate answers with the long-range context.
Key components include:
- Global Attention: Applied specifically to the question and the five candidate answers. This ensures that every token in the long context passage can "see" the question and potential answers during encoding.
- Chunking: For documents exceeding the 4,096-token limit, the context is split into chunks, processed, and the resulting logits are averaged to produce a final score.
Results and Discussion
The Longformer-based approach significantly outperformed the Gated Attention (GA) baseline and standard BERT models, achieving an approximate 40% absolute improvement over the baseline.
| Subtask | Baseline (GA) | BERT | Our Method (Longformer) |
|---|---|---|---|
| Subtask 1 (Accuracy) | 23.01% | 63.43% | 70.30% |
| Subtask 2 (Accuracy) | 22.95% | 58.76% | 64.38% |
The results indicate that a larger receptive field is essential for abstract reasoning. Subtask 2 proved more challenging, likely because extremely high-level abstractions like "entity" are harder for models to distinguish than more specific abstract concepts. The authors also noted that the current method of concatenating all five candidate answers might introduce "concatenation noise" due to position embedding interference.
Conclusion
The NLP-IIS@UT system demonstrates that combining contextualized embeddings with long-range attention is highly effective for abstract MRC. Future work will explore the integration of external knowledge bases (like WordNet) and task-specific architectures to further bridge the gap in abstract knowledge representation.
Summary updated