Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA

Ieva Staliunaite, Ignacio IacobacciEMNLP 2020 · DOI

Abstract

This study investigates the linguistic "blind spots" of popular language models—BERT, RoBERTa, and DistilBERT—within the context of conversational question answering (CoQA). By identifying systematic failures in compositional semantics (negation, semantic roles), lexical semantics (antonyms), and basic arithmetic (counting), we demonstrate that explicit injection of linguistic knowledge via multitask learning and data augmentation can significantly enhance performance. Our proposed ensemble model achieves up to a 42.1 F1 score improvement on particularly challenging question types.

Introduction

Despite the transformative impact of Large Language Models (LLMs) on natural language processing, the nature of their underlying understanding remains opaque. A critical question persists: do these models grasp the hierarchical composition of language, or do they rely on sophisticated statistical pattern matching? This research addresses this by examining the CoQA benchmark, which requires tracking dialogue and performing semantic inference across multiple sentences. We specifically target gaps in compositional semantics, lexical semantics, and basic arithmetic.

Diagnostic Analysis

To understand model limitations, we categorized CoQA questions into specific linguistic classes, such as negation (NEG), antonyms (ANT), and numerical counting (1-5).

Baseline performance of RoBERTa, BERT, and DistilBERT across various linguistic categories.

As shown in the diagnostic results, standard models exhibit significant weaknesses in specific areas. For instance, "counting" (the 1-5 class) presents a major hurdle, as extractive models are designed to select existing text spans rather than synthesize numerical answers. Furthermore, models often suffer from "negation blindness," frequently ignoring cues like "not" and incorrectly identifying negated entities as answers. Interestingly, while DistilBERT retains lexical knowledge (like antonyms) relatively well, it loses significant compositional reasoning ability compared to its larger counterparts.

Methodology

Our approach involves two primary stages: systematic error analysis followed by targeted model enhancement.

Multitask Learning (MTL)

We integrated auxiliary "heads" into the models to force the simultaneous learning of linguistic features, including negation scope, semantic roles (SRL), and sentence sentiment. This encourages the model to develop a more robust internal representation of linguistic structure.

Surprising Word Substitution

To mitigate "name bias"—where models favor common entities over rare ones regardless of context—we employed data augmentation. By replacing common entities with "surprising" ones (low-probability words in context), we forced the models to rely on syntactic structure rather than frequency-based biases.

Ensembling

We developed an ensemble strategy that selects the final answer from the specialized model variant demonstrating the highest confidence for a specific question category.

Experimental Results

The proposed enhancements yielded measurable improvements across all categories. The ensemble models provided an overall F1 score boost of 2.2 to 2.7 points over baselines.

Performance comparison of various model enhancements and the final ensemble against the RoBERTa base.

The most dramatic gains were observed in the most difficult classes. For example, the "counting" (1-5) F1 score for RoBERTa jumped from 40.1 to 82.2—a 42.1 point increase. Similarly, the "surprisal" augmentation successfully helped BERT overcome its bias toward common names, improving its ability to identify rare entities when required by the syntax.

Discussion and Conclusion

While these methods significantly improve performance, limitations remain. The extractive nature of the underlying models still restricts their ability to answer questions requiring complex mathematical synthesis. Additionally, the "surprisal" thresholding requires manual tuning for different domains.

Future research should explore integrating these linguistic features directly into the pre-training phase of larger models. This could potentially build more robust "out-of-the-box" reasoning capabilities, reducing the need for task-specific multitask learning and specialized ensembles.

Summary updated

BibTeX
@inproceedings{staliunaite2020compositional,
  author = {Ieva Staliunaite and Ignacio Iacobacci},
  title = {{Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.573},
}
Figure