TL;DR
This paper introduces Aesop, a novel platform for collaborative visual storytelling that bridges the gap between natural language and 3D animation. By integrating language parsing, gesture tracking, and a sophisticated animation engine, Aesop enables mixed-initiative AI systems to co-create narratives with humans, paving the way for more intuitive and engaging human-AI interaction.
What Problem Does This Paper Address?
Current human-AI interaction largely focuses on humans issuing directives to machines. The authors argue that the future of AI lies in mixed-initiative collaboration, where both humans and AI agents can communicate goals and operate as equals. This is particularly challenging in creative domains like storytelling, which require understanding and generating complex sequences of events, spatial reasoning, and temporal relationships. Existing platforms often fall short by being limited to simplistic environments, rigid interactions, or by not effectively integrating multimodal communication (language and gestures). Aesop aims to address this by enabling AI to understand and generate visual narratives through a combination of verbal and non-verbal cues.
Methods
Aesop is a modular system with several key components:
- Language and Gesture Parser: Utilizes the TRIPS system for semantic parsing of natural language, generating a "logical form" that represents events and their relationships. This is augmented by a gesture tracking module that uses depth and color data from a Microsoft Kinect to detect hundreds of unique human gestures.
- Composition Graphs: These graphs serve as an intermediate representation, modeling spatio-temporal and object-attribute relationships within the animation. They link actors, props, spatial relationships, and temporal events, built incrementally using natural language. Aesop uses a probabilistic graphical model for this.
- Dialogue Manager: Manages the collaborative state between the user and the system. It uses Markov Decision Processes (MDPs) to schedule actions for various "movie-making agents" (e.g., camera, director, animator). It can prompt the user for clarification when information is missing or sample from its probabilistic model to initiate communicative acts.
- Animation Software (Muvizu): This 3D animation software is used to generate the visual output. Aesop interacts with Muvizu via an API, allowing it to spawn actors, customize them, position cameras, issue animation commands, and record scenes.
Datasets
Not explicitly discussed in the paper, though the "MovieGraphs dataset" is mentioned as a source for learning parameters of the underlying probabilistic model for composition graphs.
Key Insights and Ideas
- Bridging Modalities: The core innovation is the seamless integration of natural language understanding with human gesture tracking to drive 3D animation, enabling a richer form of communication.
- Mixed-Initiative Storytelling: Aesop moves beyond directive-based AI towards a collaborative model where both human and AI can contribute to the narrative creation process.
- Spatio-Temporal Reasoning: The platform's architecture is designed to handle the complex spatio-temporal reasoning essential for visual storytelling, representing events and relationships in a structured way.
- Composition Graphs as a Universal Representation: Using composition graphs as an intermediate representation allows for flexible modeling of narrative elements and facilitates reasoning about how language and gestures translate into visual actions.
Key Results
The paper presents Aesop as a functional platform and describes its capabilities rather than reporting specific quantitative results from experiments. The key outcome is the demonstration of a system that can:
- Parse natural language and human gestures to understand narrative intent.
- Generate spatio-temporal event graphs representing story elements.
- Direct a 3D animation software (Muvizu) to create visual narratives based on these inputs.
- Enable mixed-initiative interaction for collaborative storytelling.
Limitations and Future Work
- Limitations: The paper mentions that the system's ability to handle missing information relies on prompting the user or sampling from its probabilistic model, implying potential for ambiguity or less-than-ideal choices. The rigidity of some existing platforms is highlighted as a contrast, suggesting that Aesop's own environment might still have limitations in expressiveness or complexity compared to real-world scenarios.
- Future Work: The authors suggest that Aesop will help address open research problems in conversational AI, joint language and gesture reasoning, planning and dialogue management, and narrative generation. The platform is positioned as a tool for further research in these areas.
Summary updated