Abstract
While AI learning interactions are often optimized for agent information gain, the cognitive burden on human teachers is frequently overlooked. This paper introduces a taxonomy of human-agent interactions—Showing, Categorizing, Sorting, and Evaluating—and investigates their impact on human factors. Empirical results indicate that "Evaluating" (corrections) is significantly more taxing and less usable than other methods, despite its high informativeness for the learner.
Introduction
In the field of Learning from Humans (LfH), researchers typically select interaction types, such as demonstrations or binary labels, based on how quickly they help an agent converge on a solution. However, humans are not perfect oracles; they are limited by cognitive load, fatigue, and frustration. If an interaction is too demanding, the resulting data becomes noisy or biased, ultimately degrading the agent's performance. This research addresses the lack of a principled framework for comparing these interactions from a human-centric perspective, bridging the gap between algorithmic efficiency and human factors like mental effort and usability.
A Taxonomy of Human-Agent Interaction
The researchers formalize the interaction session as a Markov Decision Process (MDP), where the human acts as the decision-maker providing feedback to the AI "environment." They categorize interactions based on four key features: query size, response size, intervention options, and response choice space. This formalization yields four primary archetypes:
- Showing: Providing full demonstrations.
- Categorizing: Assigning labels or rewards.
- Sorting: Providing preferences or rankings.
- Evaluating: Offering corrections or critiques.
Experimental Methodology
To test this taxonomy, a mixed-design study was conducted with 144 participants across two distinct domains:
- Sequential Decision Making (SDM): Utilizing a Lunar Lander game where participants either flew the craft (Showing), rated a flight (Categorizing), compared two flights (Sorting), or highlighted the "best" part of a flight (Evaluating).
- Classification: Utilizing the Pascal VOC 2012 image dataset. Participants provided their own captions, rated AI-generated captions, compared two captions, or selected image regions that justified a caption.
Cognitive load was measured objectively using a secondary task (reaction time to a color-changing circle) and subjectively via validated scales including NASA-TLX, SUS, and Paas.
Results and Discussion
The "Evaluating" Penalty
Across both domains, "Evaluating" was found to be the most frustrating, complex, and least usable interaction. In the Classification task, it was nearly twice as slow as other methods ($p < 0.01$). While corrections are highly informative for algorithms, they require humans to process a proposal, identify errors, and formulate a better alternative—a high-effort cognitive loop that significantly degrades usability scores.
The "Showing" Paradox
Theoretically, "Showing" (full demonstration) has a massive response space. However, it was consistently rated as low-load and highly usable. This suggests that humans utilize "cognitive shortcuts"; it is often easier for an expert to perform a task than to critique another's attempt.
Domain-Specific Preferences
User preference was not universal but depended on the task type. In SDM, the preference order was Showing > Sorting > Categorizing > Evaluating. In Classification, it shifted to Categorizing > Sorting > Showing > Evaluating. This suggests that the optimal interaction type depends on whether the task is more intuitive to perform or to describe.
Conclusion
The high cognitive load associated with complex feedback types like "Evaluating" can lead to decreased user confidence and potentially lower-quality data. Future work should investigate if this load results in objectively worse data for AI training and explore dynamic interaction systems that can switch archetypes based on real-time user state.
Summary updated