KARaML: Integrating Knowledge-Based and Machine Learning Approaches to Solve the Winograd Schema Challenge

Suk Joon Hong, Brandon Bennett, Judith Clymo, Lucía Gómez ÁlvarezAAAI 2022 · OpenReview

KARaML: Integrating Knowledge-Based and Machine Learning Approaches to Solve the Winograd Schema Challenge

Introduction

The Winograd Schema Challenge (WSC) was designed as a robust alternative to the Turing Test, requiring machines to resolve ambiguous pronouns in sentence pairs that differ by only one or two words (e.g., "The trophy didn't fit in the suitcase because it was too [large/small]"). Current AI approaches generally fall into two camps: Machine Learning (ML) models like BERT, which are effective but "brittle" and unexplainable, and Knowledge Representation (KR) systems, which are interpretable but difficult to scale. KARaML (Knowledge Assimilation based on Roles and Machine Learning) addresses this "scalability vs. interpretability" trade-off through an asymmetric integration of both methods.

The KARaML Framework

KARaML utilizes a hybrid pipeline that partitions the reasoning process between symbolic logic and neural networks. The system first identifies if a sentence belongs to a specific domain (e.g., situations involving "helping" or "asking"). For sentences within these domains, the framework follows a structured reasoning path.

The KARaML pipeline integrates domain filtering, symbolic extraction, and focused machine learning for semantic role matching.

As shown in the architecture above, the system uses Symbolic Extraction (via K-Parser and Answer Set Programming) to assign high-level semantic roles to candidates, such as identifying who is the "helper" and who is being "helped." Instead of asking a Language Model (LM) to solve the entire puzzle, KARaML employs Focused ML to determine which semantic role best fits the pronoun's context. For example, BERT might be asked if the phrase "because she was inexperienced" correlates more with "giving help" or "needing help." Finally, a minimal set of logical rules matches these roles to reach a final, explainable answer.

Bridging Logic and Intuition

The core innovation of KARaML is the use of semantic roles as a bridge. These roles act as a common language that both logic engines and language models can process. The authors found that BERT performs significantly better when asked to match a phrase to a role than when asked to resolve a pronoun directly. This suggests that ML models are better at "intuition" about word correlations, while symbolic logic is superior at multi-step deduction. By using KR for the structured setup and ML for the fuzzy explanation, the system avoids the knowledge acquisition bottleneck of pure logic while maintaining interpretability.

Evaluation and Results

The researchers evaluated KARaML using the expert-curated WSC273 and a domain-filtered subset of the WinoGrande dataset. The hybrid approach consistently outperformed pure ML models in the tested domains:

Test Domain BERT_LARGE BERT_WIKI_WSCR (Fine-tuned) KARaML (Our System)
"Help" Domain 57.97% 76.33% 81.64%
"Ask" Domain 51.86% 63.02% 75.93%

Beyond accuracy, KARaML provides a traceable explanation. Instead of a black-box output, the system can state it chose a candidate because they occupied a specific role that the language model identified as the most likely fit for the context.

Limitations and Future Work

Currently, KARaML's coverage is limited to the domains for which the authors have encoded manual rules. Furthermore, the system's accuracy is dependent on the performance of the initial semantic parser. Future work will focus on expanding the number of domains and exploring more advanced language models to further automate the knowledge assimilation process.

Summary updated

BibTeX
@inproceedings{hong2022karaml,
  author = {Suk Joon Hong and Brandon Bennett and Judith Clymo and Lucía Gómez Álvarez},
  title = {{KARaML: Integrating Knowledge-Based and Machine Learning Approaches to Solve the Winograd Schema Challenge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=Z60TfK8MzO},
}
Figure