The Effect of Sensor Errors in Situated Human-Computer Dialogue

Niels Schütte, John D. Kelleher, Brian Mac Namee

TL;DR

This paper investigates how errors in a robot's perception system affect human-computer dialogue. By comparing interactions with a perfectly perceiving robot versus one with simulated sensor errors, they found that perception errors significantly increase task difficulty, leading to more abandoned scenes and a higher rate of communication breakdowns. This highlights the critical role of reliable perception for natural and effective human-robot interaction.

What Problem Does This Paper Address?

As robots become more integrated into our lives and engage in natural language dialogue, their ability to accurately perceive the environment is crucial. This paper tackles the problem of how errors in a robot's perception system—its "eyes" and "ears"—impact the quality and success of human-computer dialogue. When a robot misinterprets what it sees (e.g., mistaking a red ball for a green one, or not seeing an object at all), it can lead to misunderstandings and communication breakdowns, similar to how humans struggle when their perceptions don't align. The authors aim to quantify this impact, moving beyond studies that focus solely on speech recognition or language understanding errors.

Methods

The researchers developed a simulated environment where participants interacted with an abstract robot arm via a text-based dialogue interface. The core of their methodology involved two experiments:

  • Experiment 1 (Baseline): Participants completed a series of tasks in a simulated world with a robot that had perfect perception. This established a baseline for task difficulty and dialogue performance.
  • Experiment 2 (With Errors): The same tasks were presented, but this time, the robot's perception system was intentionally degraded with simulated sensor errors. These errors included "missing object" (the robot didn't see an object) and "color misclassification" (the robot misidentified an object's color). The researchers meticulously designed these errors to affect objects that were likely to be used as landmarks in referring expressions, based on observations from Experiment 1.

They collected metrics such as the number of actions, time taken, and the rate of ambiguous or unresolved references to evaluate dialogue success and task completion.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, the authors created their own experimental setup and a custom set of "scenes" for participants to interact with. These scenes were designed specifically for the experiments, with Experiment 1 serving as a baseline and Experiment 2 introducing controlled sensor errors into these same scenes.

Key Insights and Ideas

  • Perception Errors are a Major Hurdle: The most significant finding is that even seemingly minor perception errors can substantially increase the difficulty of situated dialogue tasks.
  • Communication Breakdown Cascade: When a robot misperceives an object, it forces the human user to reformulate their requests, leading to more ambiguous or unresolved references and a higher chance of task abandonment.
  • Landmark Importance: The design choice to target objects likely to be used as landmarks proved effective. Errors affecting these key objects had a more pronounced impact on the dialogue.
  • Human Adaptability (with limits): While humans can adapt by trying different expressions, the increased effort and potential for failure highlight the system's limitations.
  • Baseline is Crucial: Experiment 1 provided essential baseline data, allowing for a direct comparison to quantify the impact of errors in Experiment 2.

Key Results

  • Increased Task Difficulty: Participants abandoned scenes with sensor errors more often than scenes without errors.
  • More Actions Required: On average, participants needed more actions to complete scenes when the robot had perception errors.
  • Higher Reference Problems: The percentage of unresolved or ambiguous references increased significantly in scenes with sensor errors. For example, in Experiment 2, the average percentage of ambiguous/unresolved references was 8.5%, compared to 6% in Experiment 1.
  • Significant Differences: Statistical tests showed that 12 out of 22 corresponding scene pairs (error vs. no error) had significantly different numbers of actions, with error scenes generally requiring more.
  • Specific Error Impact: Scene 14, designed to be uncompletable due to errors, was abandoned by all participants.

Limitations and Future Work

Limitations: * The authors acknowledge that their current analysis of unresolved/ambiguous references doesn't account for cases where a reference was resolved, but to the wrong object. This is a potential area for future work. * The study focused on specific types of errors (missing object, color misclassification). Other types of perception errors might have different impacts. * The effect of errors on completion time was less conclusive, suggesting complex user behavior that needs further investigation.

Future Work: * The researchers plan to analyze the specific strategies participants employed when encountering errors and compare them to human-human dialogue strategies. * They aim to investigate how these strategies evolve over the course of an experiment and the variation between individual participants. * A third experiment is being prepared where the system will offer different ways for participants to access the robot's understanding of its perception, such as generating descriptions of what it "sees." This will help evaluate how providing transparency into the robot's perception affects interaction.

Summary updated

BibTeX
@inproceedings{schutte2014effect,
  author = {Niels Schütte and John D. Kelleher and Brian Mac Namee},
  title = {{The Effect of Sensor Errors in Situated Human-Computer Dialogue}},
  booktitle = {VL @ COLING},
  year = {2014},
  doi = {10.3115/v1/w14-5401},
}

Loading related content...

Figure