Abstract: This paper presents the Semantic Similarity Engine (SSE), a framework designed to solve and explain multiple-choice analogy questions by navigating large-scale semantic networks like ConceptNet. Unlike traditional "black box" statistical models, SSE identifies relational parallels between word pairs and generates human-readable justifications. Evaluated on SAT and K-12 datasets, the system achieved high precision and produced explanations that human evaluators agreed with 96% of the time.
Introduction
Analogical reasoning—the ability to understand that "A is to B as C is to D"—is a cornerstone of human intelligence. While AI has made significant strides in solving these puzzles using statistical methods like Latent Relational Analysis, these approaches are typically "black boxes." They can identify the correct answer from a list but fail to explain the underlying relationship.
The challenge lies in the nature of human knowledge representation. Hand-crafted ontologies are precise but limited in scope, while massive, crowd-sourced semantic networks like ConceptNet are vast but "noisy," containing many irrelevant or inconsistent links. This research introduces a method to filter this noise, identify meaningful relational paths, and translate those paths into natural language explanations.
The Semantic Similarity Engine (SSE)
The authors introduce the Semantic Similarity Engine (SSE), which operates in three main stages to bridge the gap between raw data and human-like reasoning:
- Semantic Context Subgraph Extraction: For any word pair (e.g., goose:flock), the system performs a breadth-first search in ConceptNet to find all paths connecting the two words. It then prunes the graph to isolate the nodes that lie on a sequence between the start words, effectively capturing the "context" of their relationship.
- Sequence Similarity: To compare two word pairs, the SSE identifies the Highest Similarity Sequence Pair (HSSP). It iterates through paths of equal length in both contexts and calculates a similarity score based on the overlap of relation types such as IsA, PartOf, or UsedFor.
- Explanation Generation: Once the best matching path is found, the system selects the most salient common edge for each segment. It then uses a template-based dictionary to convert these technical relations into English sentences (e.g., converting a PartOf edge into the phrase "is a part of").
Key Insights: Navigating Indirect Paths
A critical finding of this work is that indirect paths often hold the key to an analogy. Frequently, two words are not directly linked by a clear relationship in a database. For example, goose and flock might only be directly linked by a vague "related to" tag. However, the SSE discovers that both are strongly connected to the intermediate node bird via IsA and PartOf links, revealing a clear "member-to-group" analogy.
By looking for commonalities across multiple paths rather than relying on a single "shortest path," the algorithm mitigates errors inherent in automatically generated semantic networks. Furthermore, the SSE only attempts an answer if it finds a path with a non-zero similarity score, leading to higher precision on the questions it chooses to answer.
Evaluation and Results
The study evaluated the SSE on two primary datasets totaling 600 questions:
- SAT Dataset: 373 high-difficulty questions from US college admittance tests.
- K-12 Dataset: 227 questions targeted at elementary through high school levels.
On elementary-level questions, the system achieved 85% accuracy on attempted questions. Even on the difficult SAT dataset, it maintained 40% accuracy, significantly outperforming the 20% expected by random guessing. Most importantly, in a crowdsourced study, human evaluators agreed with 96% of the explanations generated by the SSE, confirming that the system's logic aligns with human reasoning.
Limitations and Future Work
The primary limitation is coverage; the system currently fails to answer approximately 14% of questions because it cannot find a path between the words within its search limits. Future work aims to refine the natural language generation for more fluid explanations and apply this semantic path-finding approach to robotic scene understanding and automated skill transfer, where agents must recognize analogies between new and mastered tasks.
Summary updated