Inference on Syntactic and Semantic Structures for Machine Comprehension

Chenrui Li, Yuanbin Wu, Man LanAAAI 2018 · OpenReview

Abstract

This paper addresses the limitations of independence assumptions in hidden variable models for Machine Comprehension (MC). By leveraging linguistic structures—specifically dependency trees and semantic frames—the authors propose a structured inference framework that preserves grammatical and semantic relationships during word alignment. The resulting models achieve state-of-the-art accuracy on the MCTest benchmark, demonstrating that explicit structural priors are highly effective, especially in low-data regimes where deep learning models may struggle to learn complex patterns from scratch.

1. Introduction

Machine Comprehension (MC) requires a system to read a narrative and answer questions based on its content. Many existing approaches utilize "hidden variable models" to find the most relevant sentences or optimal alignments between words in a question and the source text. However, a significant gap in these models is the independence assumption, which treats hidden variables as isolated events. This often leads to linguistically inconsistent alignments—for example, a subject and its verb might be aligned to unrelated parts of the text, violating their grammatical relationship. This paper introduces a method to capture global, structural dependencies to ensure the "logic" of the question is preserved during the inference process.

2. Methodology

The authors propose two primary models that move beyond simple lexical matching by incorporating linguistic structure into the inference process.

2.1 The Tree Model

This model organizes hidden variables based on the dependency tree of the question or answer. If two words share a syntactic relationship (such as a head and its modifier), the model scores their alignments jointly. To maintain computational feasibility, the authors employ a structured inference approach—specifically a message-passing algorithm similar to those used in Markov Random Fields—to find the globally optimal alignment across the tree.

2.2 The Frame Model

The Frame Model focuses on semantic frames, which group words based on their roles in an action (e.g., a "Buying" frame includes a buyer, a seller, and an item). By grouping hidden variables according to these frames, the model ensures that the "trigger" word (the action) and its "arguments" (the actors) are aligned to the story in a way that is semantically coherent.

Graphical representation of the proposed models: (a) baseline, (b) multiple alignments, (c) Tree Model, (d) Frame Model, and (e) pairwise model.

2.3 Training and Inference

The models are trained using an Averaged Hidden Variable Perceptron. This allows the system to learn weights for various features—such as lexical overlap, cosine similarity of word vectors, and syntactic consistency—while accounting for the fact that the "true" alignments are never explicitly labeled in the training data.

3. Experiments and Results

The models were evaluated on the MCTest dataset (MC160 and MC500 subsets), which consists of fictional stories written for elementary school students. Because the stories are fictional, the system cannot rely on external world knowledge and must find answers solely within the provided text.

The proposed system achieved a top accuracy of 72.67% on the MCTest-500 dataset, outperforming previous state-of-the-art hidden variable models (which hovered around 69-70%) and contemporary deep learning approaches (such as the Parallel-Hierarchical model at 71%). The Tree Model was particularly effective on the smaller MC160 dataset, suggesting that syntactic structure serves as a highly efficient prior for reasoning when data is scarce.

4. Discussion and Conclusion

The success of this approach demonstrates that linguistic structure (syntax and semantics) provides a natural and powerful "map" for hidden variables. By following the edges of a dependency tree, the model captures long-distance dependencies that simple sliding-window techniques miss. Furthermore, an ensemble strategy combining structured models with strong lexical baselines provided a significant performance boost, indicating that structured models are excellent at "filtering" or "verifying" simpler matching techniques.

Despite these gains, limitations remain in handling questions that require counting or complex logical reasoning. Additionally, errors in coreference resolution tools can propagate through the model, leading to incorrect alignments. Future work will explore joint inference between the tree and frame models and the integration of broader external knowledge to handle more complex reasoning tasks.

Summary updated

BibTeX
@inproceedings{li2018inference,
  author = {Chenrui Li and Yuanbin Wu and Man Lan},
  title = {{Inference on Syntactic and Semantic Structures for Machine Comprehension}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=rq8Qg6U00z5},
}
Figure