KARaML: Integrating Knowledge-Based and Machine Learning Approaches to Solve the Winograd Schema Challenge
Introduction
The Winograd Schema Challenge (WSC) was designed as a robust alternative to the Turing Test, requiring machines to resolve ambiguous pronouns in sentence pairs that differ by only one or two words (e.g., "The trophy didn't fit in the suitcase because it was too [large/small]"). Current AI approaches generally fall into two camps: Machine Learning (ML) models like BERT, which are effective but "brittle" and unexplainable, and Knowledge Representation (KR) systems, which are interpretable but difficult to scale. KARaML (Knowledge Assimilation based on Roles and Machine Learning) addresses this "scalability vs. interpretability" trade-off through an asymmetric integration of both methods.
The KARaML Framework
KARaML utilizes a hybrid pipeline that partitions the reasoning process between symbolic logic and neural networks. The system first identifies if a sentence belongs to a specific domain (e.g., situations involving "helping" or "asking"). For sentences within these domains, the framework follows a structured reasoning path.
As shown in the architecture above, the system uses Symbolic Extraction (via K-Parser and Answer Set Programming) to assign high-level semantic roles to candidates, such as identifying who is the "helper" and who is being "helped." Instead of asking a Language Model (LM) to solve the entire puzzle, KARaML employs Focused ML to determine which semantic role best fits the pronoun's context. For example, BERT might be asked if the phrase "because she was inexperienced" correlates more with "giving help" or "needing help." Finally, a minimal set of logical rules matches these roles to reach a final, explainable answer.
Bridging Logic and Intuition
The core innovation of KARaML is the use of semantic roles as a bridge. These roles act as a common language that both logic engines and language models can process. The authors found that BERT performs significantly better when asked to match a phrase to a role than when asked to resolve a pronoun directly. This suggests that ML models are better at "intuition" about word correlations, while symbolic logic is superior at multi-step deduction. By using KR for the structured setup and ML for the fuzzy explanation, the system avoids the knowledge acquisition bottleneck of pure logic while maintaining interpretability.
Evaluation and Results
The researchers evaluated KARaML using the expert-curated WSC273 and a domain-filtered subset of the WinoGrande dataset. The hybrid approach consistently outperformed pure ML models in the tested domains:
| Test Domain | BERT_LARGE | BERT_WIKI_WSCR (Fine-tuned) | KARaML (Our System) |
|---|---|---|---|
| "Help" Domain | 57.97% | 76.33% | 81.64% |
| "Ask" Domain | 51.86% | 63.02% | 75.93% |
Beyond accuracy, KARaML provides a traceable explanation. Instead of a black-box output, the system can state it chose a candidate because they occupied a specific role that the language model identified as the most likely fit for the context.
Limitations and Future Work
Currently, KARaML's coverage is limited to the domains for which the authors have encoded manual rules. Furthermore, the system's accuracy is dependent on the performance of the initial semantic parser. Future work will focus on expanding the number of domains and exploring more advanced language models to further automate the knowledge assimilation process.
Summary updated