Towards Cognitive-and-Immersive Systems: Experiments in a Shared (or common) Blockworld Framework

Matthew Peveler, Biplav Srivastava, Kartik Talamadupula, Naveen Sundar G., Selmer Bringsjord, Hui SuarXiv 2017 · OpenReview

Abstract

As computational power has continued to increase, and sensors have become more accurate, the corresponding advent of systems that are at once cognitive and immersive has arrived. These cognitive and immersive systems (CAISs) fall squarely into the intersection of AI with HCI/HRI: such systems interact with and assist the human agents that enter them, in no small part because such systems are infused with AI able to understand and reason about these humans and their knowledge, beliefs, goals, communications, plans, etc. We herein explain our approach to engineering CAISs. We emphasize the capacity of a CAIS to develop and reason over a 'theory of the mind' of its human partners. This capacity entails that the AI in question has a sophisticated model of the beliefs, knowledge, goals, desires, emotions, etc. of these humans. To accomplish this engineering, a formal framework of very high expressivity is needed. In our case, this framework is a cognitive event calculus, a particular kind of quantified multi-operator modal logic, and a matching high-expressivity automated reasoner and planner. To explain, advance, and to a degree validate our approach, we show that a calculus of this type satisfies a set of formal requirements, and can enable a CAIS to understand a psychologically tricky scenario couched in what we call the cognitive polysolid framework (CPF). We also formally show that a room that satisfies these requirements can have a useful property we term expectation of usefulness. CPF, a sub-class of cognitive microworlds, includes machinery able to represent and plan over not merely blocks and actions (such as seen in the primitive 'blocks worlds' of old), but also over agents and their mental attitudes about both other agents and inanimate objects.

Introduction

Most AI decision-support systems are designed for a single user. However, human problem-solving is inherently social, involving groups with differing levels of knowledge, conflicting goals, and evolving beliefs. While "intelligent rooms" have existed for decades, they typically function as sophisticated interfaces for data retrieval rather than active cognitive partners. This paper introduces a formal framework for Cognitive and Immersive Systems (CAISs)—intelligent environments designed to assist human groups by reasoning about their mental states.

The Formal Framework: DCEC

To build a system that "thinks" about what its users are thinking, the researchers employ a logic-based approach centered on the Deontic Cognitive Event Calculus (DCEC). This is a highly expressive, quantified multi-operator modal logic. Unlike standard first-order logic, DCEC includes operators for knowledge ($K$), belief ($B$), desire ($D$), and common knowledge ($C$), allowing the system to represent complex nested thoughts (e.g., "The room believes that Alice believes Bob is unaware of the change").

Table of sorts used in the formal framework to define agents, actions, and states.

The system utilizes ShadowProver, a high-speed automated theorem prover that handles modal logic efficiently, and Spectra, a specialized planner that allows for goals defined by arbitrary DCEC formulas, such as ensuring one agent knows the state of another agent's knowledge.

System Architecture: The Mind-of-Room

The CAIS architecture is defined by a cyclical flow of information that allows the environment to act as a non-localized cognitive agent. This "Mind-of-Room" collects and analyzes context to facilitate human-scale interaction.

The Mind-of-Room architecture showing the cycle of perception, interpretation, and presentation.

The cycle consists of:

  1. Perception: Capturing and recognizing human actions via sensors like microphones and Kinects.
  2. Interpretation: Reasoning via ShadowProver and the "Society of Cognitive Agents" to understand context and mental states.
  3. Presentation: Communicating and presenting information through multi-modal displays to assist the human participants.

Experimental Validation: The Cognitive Polysolid Framework (CPF)

The authors evaluate their system using the Cognitive Polysolid Framework (CPF). While inspired by the classic "blocks world," CPF is a "cognitive microworld" that populates the environment with agents possessing mental attitudes. In a typical scenario, the system must track when an agent develops a false belief—for instance, if Agent A moves a block while Agent B is out of the room.

Key findings from these experiments include:

  • Intensional Logic Necessity: The authors prove that standard extensional logic leads to unsound inferences regarding mental states. Modal operators are required to correctly model that an agent might know a person's role without knowing their specific identity.
  • Expectation of Usefulness: A foundational property where a correctly functioning CAIS allows an agent to expect the room to intervene when it perceives another agent is misinformed.
  • Real-time Performance: The system tracked beliefs and generated intervention plans with an average latency of only 748ms.

Conclusion and Future Work

This research demonstrates that a CAIS can successfully track "Theory of Mind" to rectify missing or false beliefs among human participants in real-time. Future work aims to scale these logic-based systems beyond microworlds into complex domains like business negotiations and to incorporate social choice theory to mediate conflicting group priorities.

Summary updated

Abstract

As computational power has continued to increase, and sensors have become more accurate, the corresponding advent of systems that are at once cognitive and immersive has arrived. These \textit{cognitive and immersive systems} (CAISs) fall squarely into the intersection of AI with HCI/HRI: such systems interact with and assist the human agents that enter them, in no small part because such systems are infused with AI able to understand and reason about these humans and their knowledge, beliefs, goals, communications, plans, etc. We herein explain our approach to engineering CAISs. We emphasize the capacity of a CAIS to develop and reason over a `theory of the mind' of its human partners. This capacity entails that the AI in question has a sophisticated model of the beliefs, knowledge, goals, desires, emotions, etc.\ of these humans. To accomplish this engineering, a formal framework of very high expressivity is needed. In our case, this framework is a \textit{cognitive event calculus}, a particular kind of quantified multi-operator modal logic, and a matching high-expressivity automated reasoner and planner. To explain, advance, and to a degree validate our approach, we show that a calculus of this type satisfies a set of formal requirements, and can enable a CAIS to understand a psychologically tricky scenario couched in what we call the \textit{cognitive polysolid framework} (CPF). We also formally show that a room that satisfies these requirements can have a useful property we term \emph{expectation of usefulness}. CPF, a sub-class of \textit{cognitive microworlds}, includes machinery able to represent and plan over not merely blocks and actions (such as seen in the primitive `blocks worlds' of old), but also over agents and their mental attitudes about both other agents and inanimate objects.
BibTeX
@misc{peveler2017towards,
  author = {Matthew Peveler and Biplav Srivastava and Kartik Talamadupula and Naveen Sundar G. and Selmer Bringsjord and Hui Su},
  title = {{Towards Cognitive-and-Immersive Systems: Experiments in a Shared (or common) Blockworld Framework}},
  year = {2017},
  eprint = {1709.05958},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.05958},
}
Figure