TL;DR
Researchers have developed the first Portuguese-language version of the Winograd Schema Challenge (WSC), a benchmark designed to test a machine's commonsense reasoning through ambiguous pronoun resolution. By adapting 277 schemas and testing them against a baseline LSTM language model, the authors demonstrate that while the task is significantly harder in gendered languages, current small-scale models still struggle to beat random chance.
What Problem Does This Paper Address?
The Winograd Schema Challenge (WSC) was proposed as a more rigorous alternative to the Turing Test. It consists of sentence pairs that differ by only one "special word," which changes which antecedent a pronoun refers to (e.g., "The trophy doesn't fit into the brown suitcase because it is too large/small"). To solve this, a machine can't rely on simple statistics; it needs "commonsense" understanding of the world.
While WSC is a staple in English NLP, it hasn't been widely available for Portuguese. Furthermore, Portuguese presents a unique linguistic hurdle: grammatical gender. In English, "it" is gender-neutral. In Portuguese, nouns are masculine or feminine. If a sentence has one masculine and one feminine candidate, the pronoun's gender would give the answer away instantly, destroying the "challenge" of the schema.
Methods
The authors took a two-pronged approach:
- Dataset Adaptation: They manually translated 285 English schemas. To maintain the difficulty, they had to ensure both candidate antecedents shared the same grammatical gender. This often required creative adaptation—for instance, changing "trophy" (masculine) to "medal" (feminine) to match "suitcase" (feminine).
- Baseline Solver: They implemented a "substitution" method using a Language Model (LM). The system replaces the ambiguous pronoun with each candidate antecedent and calculates the joint probability of the resulting sentences. The sentence with the higher probability is chosen as the "correct" interpretation.
- Architecture: The baseline uses a relatively simple neural network: two LSTM layers with 200 hidden units and an embedding size of 200.
Datasets
- Portuguese WSC: A new set of 277 Winograd Schemas (8 were discarded from the original 285 due to translation difficulties).
- Training Corpus: A 15MB subsection of a Portuguese Wikipedia dump (approx. 2 million tokens).
- English Comparison: To provide a fair baseline, they also trained an identical model on the Wikitext-2 dataset to test against the original English WSC.
Key Insights and Ideas
- The Gender Trap: Direct translation is impossible for WSC. In Portuguese, if you say "The trophy (M) doesn't fit in the suitcase (F) because he is too large," the grammar solves the puzzle for you. The authors' "aha moment" was realizing they had to swap nouns for gender-matched synonyms to preserve the logic puzzle.
- Grammatical Friction: Automatic substitution (replacing a pronoun with a noun) often breaks Portuguese grammar, specifically contractions (e.g., "através dela" becomes the ungrammatical "através a fenda" instead of "através da fenda").
- The "Good Enough" Baseline: Surprisingly, manually fixing these grammatical errors in the test set didn't significantly improve the model's performance. This suggests that for small models, the lack of world knowledge is a much bigger bottleneck than minor grammatical hiccups.
Key Results
The results highlight just how high the bar is for commonsense reasoning:
- Portuguese Accuracy: The baseline model achieved roughly 45.13% accuracy on the full dataset.
- English Accuracy: An identical model achieved 50.55% on the English set.
- Consistency: The model was only consistent (giving the correct answer for both versions of a schema pair) about 9-13% of the time.
Since a random guess would yield 50%, these results show that simple LMs trained on small corpora are essentially guessing. This sets a clear baseline for future researchers using larger models like BERT or GPT.
Limitations and Future Work
The primary limitation is scale. The authors used a model with 7.3 million parameters trained on 15MB of text; for comparison, state-of-the-art English models often use billions of parameters and terabytes of data.
Future work includes:
- Scaling up to larger architectures like BERT specifically pre-trained for Portuguese.
- Developing an "extended" collection of "relaxed" schemas (easier versions) to help train models.
- Testing the impact of anonymizing proper names to see if models are being distracted by specific entities rather than focusing on the logic of the sentence.
Summary updated