Operationalizing Operational Logics: Semiotic Knowledge Representations for Interactive Systems

Joseph C. OsbornIJCAI 2017 · DOI

TL;DR

This paper proposes a formal framework called "operational logics" to represent how interactive systems, especially games, communicate their internal workings to users and designers. By bridging the gap between abstract system mechanics and their user-facing semiotics, this approach enables more robust AI analysis and development for interactive systems.

What Problem Does This Paper Address?

Developing AI for interactive systems, like video games, presents unique challenges beyond traditional AI problems. While we can model the functional requirements of these systems (e.g., state transitions, physics), we also need to consider their "semiotic" aspects – how they are legible and understandable to human users and designers. Current approaches to this are often ad hoc, leading to a disconnect between powerful AI analysis tools and the complex, interactive nature of games. This paper aims to create a unified approach that formally captures both the underlying mechanics and the user-facing communication strategies of interactive systems, making them amenable to rigorous analysis.

Methods

The core contribution is the refinement and formalization of "operational logics" (OLs). An OL combines two key components:

  1. Abstract Process: The underlying mechanics of the interactive system (e.g., agent behaviors, physics simulations, resource management).
  2. Communicative Strategy: How the system presents its abstract process to the user (e.g., visual feedback, projections, animated readouts).

The authors propose finding formal logical theories that correspond to these operational logics, allowing for compositional modeling and analysis. They leverage existing formalisms like hybrid automata (which combine finite state machines with differential equations) to represent these systems. The work involves enumerating common OLs and their corresponding formal logics, and developing tools for learning these representations from observed behavior.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Semiotics are Crucial for Interactive AI: Interactive systems aren't just about functional correctness; their legibility and how they communicate their state and actions to humans are paramount.
  • Bridging the Gap: Operational logics provide a formal bridge between the abstract, computational aspects of a system and its user-facing semiotic representation.
  • Formalizing the "Game Feel": By operationalizing OLs, we can move beyond ad hoc methods to formally model and analyze aspects like "game feel" and user experience.
  • Enabling Compositional Analysis: A formal, unified framework for OLs allows for compositional analysis of complex interactive systems, breaking them down into understandable components.
  • AI for Understanding AI: This work enables AI to operate at both the level of underlying models and presented information, facilitating tasks like reverse-engineering and automatic documentation.

Key Results

The paper describes ongoing and completed projects that demonstrate the utility of operationalizing OLs:

  • HyPED: A modeling language and model checker for games based on hybrid automata, enabling analysis of entity-state, physics, and collision logics.
  • Learning Hybrid Automata: Work on learning parameters (flows, reset values) of hybrid automata with known structures, and learning both structure and flows from observed behavior (CHARDA). These projects show improved specification recovery by considering human interaction and simulated object behavior.

Limitations and Future Work

The authors acknowledge several areas for future work:

  • Hybrid Planning: Exploring continuous-time techniques for hybrid planning that don't rely on non-linear programming.
  • Connotative Semiotics: Capturing more of the connotative visual meaning in games (e.g., what "spiky" or "burning" implies beyond literal representation).
  • Compositional and Modular Game Analysis: Developing techniques for compositional analysis and automated abstraction of games.
  • Broader Domains: Extending the application of operational logics to a wider range of games and interactive systems.
  • Integration with AI Community: Becoming better integrated with the broader AI community and identifying ways to influence AI outside of games.

Summary updated

BibTeX
@inproceedings{osborn2017operationalizing,
  author = {Joseph C. Osborn},
  title = {{Operationalizing Operational Logics: Semiotic Knowledge Representations for Interactive Systems}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/759},
}
Figure