Learning from the Absence of Human Feedback

This direction is merged into another direction: implicit feedback interpretation. Its last version stays readable and citable here.

Version 1 of 116 papers30 conceptsfrozen 2026-10-04

In traditional Reinforcement Learning (RL), an agent learns by maximizing a numerical reward signal. When we put a human in the loop to "teach" the agent, we often treat the human like a biological reward function: if the human clicks a "plus" button, the agent gets +1; if they click "minus," it gets -1.

However, human teachers are not buttons. They are strategic communicators. Most of the time, a human teacher is silent. In a standard RL framework, silence is treated as a zero—a neutral signal that provides no information. But as researchers like Thomas Cederborg and Charles Lee Isbell Jr. argue in Policy Shaping with Human Teachers, this is a massive waste of data. In human interaction, silence is often "informative." If a teacher only intervenes when you make a mistake, their silence is a ringing endorsement of your current path.

Silence as a Strategic Signal

The core intuition behind this research direction is that human feedback is a communication channel, not just a scalar value. Robert Loftin and James MacGlashan formalized this in their work on SABL (Strategy-Aware Bayesian Learning).

Instead of asking "What reward did the human give?", SABL asks "Given the teacher's strategy, what does their feedback (or lack thereof) tell me about my action?" If a teacher follows a "praise-on-success" strategy, silence implies the action was wrong. If they follow a "critique-on-error" strategy, silence implies the action was right. By modeling the probability of silence, Loftin showed that agents can learn target behaviors much faster because they are essentially receiving a signal at every single time step, even when the teacher isn't doing anything.

The Zero-Cost Data Stream

Beyond intentional silence, humans provide a wealth of "passive" feedback. In the EMPATHIC framework, Yuchen Cui and Peter Stone demonstrate that agents can learn from involuntary human reactions, such as facial expressions.

This shifts the burden of teaching. In traditional "active" feedback, the human must exert cognitive effort to evaluate and reward the agent. In "implicit" feedback systems, the agent "reads the room." As Yuchen Cui and Reid G. Simmons note in their survey of human-in-the-loop ML, integrating these passive channels—gaze, gestures, and silence—allows for a "zero-cost" data stream that doesn't increase the teacher's workload.

Disambiguating the Quiet Periods

The challenge with silence is that it is ambiguous. Does silence mean "Good job," or does it mean "I'm not paying attention"?

To solve this, researchers have moved toward "uncertainty-aware" models. Carlos Celemin and Jens Kober developed ICREATe, which treats silence as a positive signal only after a certain "reaction time" threshold has passed. If the robot performs an action and the human doesn't intervene within a second, the robot assumes it's on the right track.

Similarly, Haipeng Chen and Bo An introduced ABLUF to handle "messy" human feedback. Their model explicitly accounts for the fact that a human might be confused or neutral. By treating silence as its own distinct state in a probabilistic model, the agent avoids making false assumptions during periods of trainer inattention.

From Corrections to Implicit Endorsement

This paradigm is particularly powerful in robotics and interactive imitation learning. In systems like CLIC, described by Zhaoting Li and Robert Babuska, the absence of a supervisor's intervention is modeled as a signal of success.

This aligns with how humans actually teach: we stop correcting once the student has mastered the task. By interpreting the lack of correction as a reinforcement signal, agents can stabilize their policies and improve sample efficiency. As Alekh Agarwal and Teodor V. Marinov suggest in the context of contextual bandits, this "user-triggered supervision" allows models to infer that a suggestion was acceptable simply because the user didn't bother to change it.

Ultimately, this research direction moves AI away from being a passive recipient of numbers and toward being an active interpreter of human behavior. By understanding that silence is a choice, agents can learn more from what we don't say than from what we do.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_learning_from_the_absence_of_human_feedb_0ad8cc42,
  title = {Learning from the Absence of Human Feedback},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-from-the-absence-of-human-feedback/rdr_0ad8cc42d5fe48e8bcb7074aa8412246},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_from_the_absence_of_human_feedb_0ad8cc42_v1,
  title = {Learning from the Absence of Human Feedback},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-from-the-absence-of-human-feedback/rdr_0ad8cc42d5fe48e8bcb7074aa8412246/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}