TL;DR
This paper introduces a novel method for Referring Expression Generation (REG) in human-robot collaboration by leveraging shared knowledge of past activities. This extension allows robots to generate more descriptive and unambiguous references, even in situations where current environmental cues are insufficient. The approach integrates past actions into an ontology-based knowledge base, enabling richer communication and more robust human-robot interaction.
What Problem Does This Paper Address?
In human-robot collaboration, robots often need to refer to objects or entities within a shared environment. This is known as the Referring Expression Generation (REG) problem. Current REG methods typically rely solely on the immediate, current state of the environment. However, this can lead to ambiguity, especially when multiple similar objects exist or when the robot's perspective is limited. For instance, if a robot needs to refer to a specific knife, but there are several identical knives, simply describing its current properties might not be enough. This paper addresses this limitation by proposing an extension to REG that incorporates the history of human-robot collaborative activities. By remembering and utilizing past interactions, the robot can generate more precise references, such as "the knife you used to cut the tomato," which is far more informative than just "the blue knife."
Methods
The core of this paper is an extension to an existing ontology-based REG algorithm. The original algorithm used a Uniform Cost Search (UCS) to find a set of relations that uniquely identify a target entity. The novel contribution lies in augmenting the knowledge base with a Hierarchical Execution Trace (HET), which semantically represents past human-robot collaborative activities.
Here's a breakdown of the key methodological aspects:
- Ontology-based Knowledge Representation: The system uses an ontology to store knowledge, comprising a semantic knowledge base (Ks) and an episodic knowledge base (KE). Ks stores facts about entities and their properties, while KE stores temporal information about executed tasks.
- Hierarchical Task Network (HTN) Representation: The paper formalizes HTN planning domains and their execution traces (HETs) within the ontology. This allows for a structured representation of tasks, their decompositions, and the specific instances of these tasks that were performed.
- Modified REG Algorithm: The original UCS-based REG algorithm is extended. The
CREATEADDITIONfunction, responsible for generating new candidate referring expressions, is enhanced with: COMPLETIONADDITIONS: Ensures that all parameters of a task involved in a potential reference are included.ACTINGADDITIONS: Explores tasks in which the entities involved in the candidate RE are participants. This allows the robot to refer to an entity through its involvement in a past action.- Cost Function: The cost of adding a past task to a referring expression is weighted by its recency, reflecting the idea that more recent actions are easier for humans to recall.
Datasets
The paper does not introduce new datasets. Instead, it uses a pre-existing, realistically-sized knowledge base (101 entities, 36 classes, 40 properties, and 497 relations) that describes an apartment environment. This knowledge base was previously used in [4] and was augmented with task descriptions for the evaluation of the extended REG algorithm.
Key Insights and Ideas
- Past Actions as Powerful Referents: The most significant insight is that past collaborative actions provide a rich and often disambiguating source of information for referring expressions, especially when current environmental context is insufficient.
- Bridging the Gap Between Planning and Communication: By integrating HTN execution traces into an ontology, the system effectively bridges the gap between task planning and natural language communication, allowing robots to leverage their "memory" of joint activities.
- Domain-Independent Applicability: The approach is designed to be usable with a domain-independent ontology, suggesting its potential for broad application across various HRI scenarios.
- Theory of Mind Implications: While not explicitly a "theory of mind" system, the ability to estimate a human partner's knowledge of past events implicitly touches upon understanding what the human might recall.
Key Results
The paper demonstrates the effectiveness of the proposed method through several test cases based on a simulated meal preparation scenario.
- Improved Ambiguity Resolution: In scenarios where the original REG algorithm failed due to ambiguity (e.g., multiple identical knives), the extended algorithm successfully generated unambiguous referring expressions in four out of five test cases.
- Example Verbalizations: The paper provides concrete examples of generated referring expressions, such as "the knife with which Alex prepared the salad vegetables" and "the knife with which Alex cut the tomato."
- Performance Impact: While the extension introduces some computational overhead, the performance impact is deemed acceptable for HRI. For a complex entity, the execution time increased from 6.25ms to 245.25ms in a worst-case scenario (where added tasks were not useful for REG). However, the authors argue that this is still within acceptable limits for human interaction flow and that in more realistic cases, the use of past tasks can actually reduce execution time by finding shorter, more direct references.
Limitations and Future Work
- Forgetting Mechanism: The current implementation does not include a forgetting mechanism. The authors acknowledge that this would be necessary to prevent the knowledge bases from growing indefinitely and to better model human forgetting. This is identified as future work.
- "Short" Interactions: The current focus is on "short" interactions (few hours). Extending the system to handle longer-term memory and forgetting would be a natural progression.
- Linguistic Realization: The paper focuses on the content determination of referring expressions, not the linguistic realization (i.e., turning the structured expression into natural language sentences). While the ontology ensures concepts have verbalizable labels, the actual generation of fluent language is outside the scope of this work.
- Planner Integration: The authors mention integrating the original algorithm into a cost-based human-aware task planner. Future work includes devising a planner that uses this new capability to generate "future past tasks" contexts, suggesting a proactive approach to communication.
Summary updated