Picnics, Kittens And Wigs: Using Scenarios For The Sentence Competition Task

TL;DR

This paper introduces a novel approach to sentence completion, a task crucial for reading comprehension, by leveraging "scenarios" – structured representations of common events and their components. By modeling these scenarios hierarchically and incorporating "weights" to signify the strength of associations between actions and events, the system aims to infer missing words in sentences, mimicking human-like contextual understanding.

What Problem Does This Paper Address?

The paper tackles the challenge of computer-based reading comprehension, specifically focusing on the "sentence completion task." This task, commonly found in standardized tests, requires filling in a blank in a sentence with a word that makes the most sense in context. While humans effortlessly use their "common sense" knowledge and understanding of the world to solve this, computers struggle. The authors aim to bridge this gap by developing a system that can represent and utilize this implicit knowledge, moving beyond simple syntactic analysis to a deeper semantic understanding of events.

Methods

The core of the proposed method is the use of scenarios. These are hierarchical knowledge structures that describe complex actions (e.g., "going on a picnic," "going shopping"). Each scenario contains:

  • General Information: Likely actors, typical settings, and temporal aspects of an event.
  • Specific Details: How the action is carried out.
  • Free Variables: Placeholders for objects, actors, or locations (e.g., X for the actor, Z for the destination).
  • Embedding: Scenarios can be nested within others (e.g., "going grocery shopping" is embedded within "going on a picnic"). This allows for more complex reasoning.

Key to the system's operation are:

  • Weights: Each action within a scenario has two types of weights:
  • Upward Weight (Evoking Strength): Measures how likely an action is to trigger a particular scenario.
  • Downward Weight (Frequency Strength): Measures how frequently an action occurs when a scenario is known to be happening.
  • Production Rules: Scenarios are represented as a series of rules, e.g., (observation weight) <- (scenario).
  • Synthesis: A process to avoid duplicating scenario patterns and to aggregate weights from multiple evoking actions.
  • Dominance Principle: Used to partition scenarios into "major" and "minor" lists, identifying the most probable scenario and its alternatives.
  • Blank Bindings Stack: Stores potential fillers for the blank based on scenario variable bindings.

Datasets

The paper does not explicitly mention specific datasets used for training or evaluation in the traditional machine learning sense. Instead, it uses examples from standardized reading comprehension tests, such as the Stanford Achievement Test and Gates Basic Reading Exams, to illustrate the sentence completion task and to test its proposed system. The examples provided (E1-E7) serve as the "data" for demonstrating the system's capabilities and limitations.

Key Insights and Ideas

  • Scenarios as the Foundation of Common Sense: The paper argues that "common sense" knowledge, often assumed in language tasks, can be explicitly represented through structured scenarios. This is a significant departure from purely statistical or rule-based approaches.
  • Hierarchical Knowledge Representation: Organizing scenarios in a hierarchy (e.g., "going places" as a super-scenario for "picnics," "shopping trips," etc.) allows for efficient knowledge sharing and inference. Actions true of a super-scenario automatically apply to its sub-scenarios.
  • The Power of Embedded Scenarios: Embedding scenarios (like "grocery shopping" within "picnic") is crucial for understanding the full context and inferring missing information. It allows the system to leverage knowledge from related, but distinct, events.
  • Weights as Measures of Association: The use of upward and downward weights provides a quantitative way to model the strength of relationships between actions and events, mimicking human intuition about likelihood and frequency.
  • Abduction as a Core Reasoning Process: The paper links the process of inferring a picnic from clues like "sunny day" and "food in baskets" to abduction, a form of logical inference where one infers the most likely explanation for an observation.

Key Results

The paper demonstrates the system's ability to correctly fill blanks in several examples (E3, E5, E6). For instance:

  • In example E3 ("Bob and Betty were happy as they piled baskets of good things to eat in the car. were going on a ______."), the system correctly infers "picnic."
  • In example E5 (a modified E3), it correctly identifies "Bob and Betty, Mother and Father" as the actors for the picnic.
  • It also correctly identifies "wig" and "library" for example E6.

However, the paper also highlights failures, such as in example E7, where the system fails to produce the expected results, indicating limitations in its current implementation.

Limitations and Future Work

The authors acknowledge several limitations and suggest avenues for future research:

  • Weight Tuning: The effectiveness of the system relies on carefully tuning the weights. Determining optimal values and whether to use linear or non-linear aggregation remains an open problem. An alternative proposed is to rely solely on the number of facts a scenario can explain, rather than weights.
  • Limited Causal Link Depth: The current implementation of "minor lists" restricts causal links between embedded scenarios to a depth of one. This prevents the system from finding deeper causal relationships, as seen in the failure with example E7.
  • Scenario Synthesis Limitations: The synthesis process currently does not allow scenario patterns with different time designations (e.g., initiation vs. performance) to link to a common parent node. This could lead to missing broader contextual understanding.
  • Initial Sentence Processing: The phase of reducing input sentences into a structured "observation list" was hand-simulated and requires further implementation.
  • Pronoun Resolution: While the system can infer the correct scenario, filling blanks with pronouns (like "they") requires a post-processor with specific heuristics.

Future work could focus on addressing these limitations to create a more robust and comprehensive system for understanding natural language context.

Summary updated

BibTeX
@inproceedings{1975picnics,
  title = {{Picnics, Kittens And Wigs: Using Scenarios For The Sentence Competition Task}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1975},
}
Figure