Abstract
The recently introduced BERT model exhibits strong performance on several language understanding benchmarks. In this paper, we describe a simple re-implementation of BERT for commonsense reasoning. We show that the attentions produced by BERT can be directly utilized for tasks such as the Pronoun Disambiguation Problem and Winograd Schema Challenge. Our proposed attention-guided commonsense reasoning method is conceptually simple yet empirically powerful. Experimental analysis on multiple datasets demonstrates that our proposed system performs remarkably well on all cases while outperforming the previously reported state of the art by a margin. While results suggest that BERT seems to implicitly learn to establish complex relationships between entities, solving commonsense reasoning tasks might require more than unsupervised models learned from huge text corpora.
Introduction
Commonsense reasoning remains a significant hurdle in natural language processing. A primary test for this capability is the Winograd Schema Challenge (WSC), which consists of pairs of sentences that differ by only one or two words and contain an ambiguous pronoun. For example: "The trophy doesn't fit in the suitcase because it is too small." Resolving whether "it" refers to the trophy or the suitcase requires understanding physical properties and spatial relationships, rather than just linguistic patterns.
Historically, solving these tasks required hand-crafted knowledge bases or complex supervised learning. This paper investigates whether the internal attention mechanisms of BERT, a model trained solely on unsupervised language modeling, can resolve these coreferences without any task-specific fine-tuning or external knowledge bases.
Methodology: Maximum Attention Score (MAS)
The authors propose the Maximum Attention Score (MAS), a remarkably simple unsupervised method that leverages BERT's self-attention layers. Instead of training a classifier on top of BERT's embeddings, MAS extracts the attention weights between a reference pronoun and its potential candidates directly from the pre-trained model.
The process involves:
- Extracting the attention tensor across all layers and heads of a pre-trained
bert-base-uncasedmodel. - Isolating the attention weights between the pronoun (e.g., "it") and candidate nouns (e.g., "trophy", "suitcase").
- Applying a masking operation to find the maximum attention values linking the pronoun to each candidate across the entire network.
- Summing and normalizing these maximum values to produce a final score. The candidate with the highest MAS is predicted as the referent.
Experimental Results
The MAS method was evaluated on two benchmarks: the Pronoun Disambiguation Problem (PDP-60) and the Winograd Schema Challenge (WSC-273).
On PDP-60, the model achieved 68.3% accuracy, outperforming previous unsupervised models (~60%) and even the best supervised systems (66.7%) that utilized multiple external knowledge bases. On the more difficult WSC-273, the model achieved 60.3% accuracy. This represents a significant improvement over the previous state-of-the-art of 57.1%, as shown in the table below.
Discussion and Conclusion
The success of MAS suggests that bidirectional language models like BERT implicitly learn to resolve complex coreferences during unsupervised training. Attention heads naturally evolve to link pronouns to their correct nouns, acting as a proxy for semantic understanding.
However, the paper's title—a play on the famous "Attention Is All You Need"—serves as a reminder of the limitations of purely statistical models. While BERT is adept at finding associations within a text, true commonsense reasoning often requires background knowledge that is never explicitly stated. If the necessary context is missing from the training corpus, even the most sophisticated attention mechanisms will fail. Future work may involve integrating external knowledge or fine-tuning attention specifically for reasoning tasks, but the ultimate goal of "true" commonsense likely requires moving beyond text-only modeling.
Summary updated