When to Assist and What to Suggest: Joint Gating and Memory-Grounded Hint Generation in One Compact Decoder

A. J. A. Jubayer Talukder, Mohammad Ariful HaquearXiv 2026 · arXiv

TL;DR

Researchers have developed a unified 1B-parameter decoder for proactive AI assistants that decides both when to help and what to say in a single forward pass. By using an "early-exit" mechanism, this compact model matches the performance of more complex two-model pipelines while running up to 1.76x faster on standard hardware.

What Problem Does This Paper Address?

Proactive conversational assistants—like AI-powered smart glasses—are designed to offer helpful hints (e.g., a forgotten name or date) without being asked. This requires the system to solve two intertwined problems simultaneously: gating (deciding if the current moment warrants an interruption) and generation (deciding what the hint should be).

Existing systems typically split these tasks between two separate models: a small, efficient "gate" and a larger, more capable "generator." However, this approach is computationally expensive for wearable devices and makes it difficult to optimize the system as a whole. Furthermore, most research evaluates these components in isolation, which doesn't reflect how they perform together in the real world.

Methods

The authors propose a Unified Early-Exit Decoder architecture. Instead of two models, they use a single 1B-parameter model (Llama-3.2-1B) equipped with a shared LoRA adapter.

  1. Intermediate Gating: A classifier head is attached to an intermediate layer of the decoder (e.g., layer 6 or 10). During the forward pass, the model calculates the probability that a hint is needed at that specific moment.
  2. Early Exit: If the probability is below a certain threshold, the model "exits" early, stopping all further computation for that moment to save power and time.
  3. Joint Generation: If the gate "fires," the same forward pass continues through the remaining layers to generate a short, 1–3 word hint grounded in the user's memory profile.
Comparison between the traditional two-model cascade and the proposed unified early-exit decoder.

Datasets

  • Primary Corpus: A semi-synthetic dataset consisting of 28,058 decision points across 866 dialogues, derived from the LlamaPIE, SODA, and PerLTQA corpora. It includes "memory profiles" for users to ground the hints.
  • ICSI Meeting Corpus: A secondary dataset of naturally recorded multi-party meetings used to test the gating mechanism's performance on real-world, non-synthetic conversational data.

Key Insights and Ideas

  • Memory is the Main Driver: The user's persistent memory profile (e.g., knowing the user attended a specific dance performance) is far more important for hint quality than model scale. Removing the memory profile dropped performance by nearly 25 exact-match points, while upgrading from a 1B to an 8B generator provided almost no benefit.
  • Gating is a Prefix of Generation: The authors demonstrate that the decision to speak can be effectively "read" from the intermediate layers of the generator itself. A separate gating model isn't strictly necessary if the generator is properly tuned.
  • The "Oracle-Gate" Ceiling: The paper highlights that stage-wise generation scores (evaluating the generator only on moments it should speak) represent an unreachable ceiling. Real-world performance is always lower because the gate will inevitably miss some moments or fire incorrectly (false alarms).

Key Results

  • Speed and Efficiency: The unified model achieved a 1.31x to 1.76x speedup in wall-clock time compared to a matched two-model pipeline.
  • Performance Parity: On automatic end-to-end measures, the unified 1B model matched or slightly exceeded the performance of the two-model 1B pipeline.
  • Human Validation: A human evaluation involving 1,144 ratings confirmed that hints generated with a memory profile were significantly more useful (an average "uptake" score of 4.01 vs. 3.18 without memory).
  • Metric Tracking: The study found that "LLM-as-a-judge" metrics and graded scores (like ROUGE or BERTScore) track human preferences much more closely than strict "Exact Match" scores.
End-to-end exact match performance across different gate thresholds.

Limitations and Future Work

  • Synthetic Labels: The primary dataset relies on labels generated by a larger language model rather than actual human needs in the moment. The authors addressed this partially with the ICSI corpus, but more naturalistic data is needed.
  • Text-Only Evaluation: The system was evaluated on text transcripts. In a real wearable deployment, speech-to-text errors would likely impact performance.
  • Single-Rater Human Eval: Much of the human evaluation data came from a single primary rater, though cross-validation with other raters showed good agreement.
  • Future Directions: The authors suggest exploring how these systems handle non-user consent and privacy, as a proactive assistant is essentially "always listening" to everyone in the room.

Summary updated

Abstract

Proactive assistants offer short, memory-grounded hints during a conversation, deciding when to assist and what to suggest. Current systems split this between a gate and a larger generator; we evaluate both together on 28,058 decision points. Stage-wise generation scores are oracle-gate ceilings: behind a real gate, end-to-end exact match is 0.2619 against a ceiling of 0.2971, and end-to-end precision is 0.2387, leaving substantial headroom. A gate pays off through precision and compute, whose saving falls from 4.39x with an 8B generator to 1.38x with a 1B one. The memory profile drives hint quality: removing it costs 24.7 exact-match points (rescaled BERTScore 0.57 to 0.35), while retrieval and a larger generator add nothing on exact match. One 1B decoder can do both, gating from an intermediate layer and generating in the same pass. At its final checkpoint it matches or exceeds the two-model pipelines on automatic end-to-end measures with 1.31-1.76x less wall-clock time than the matched 1B pipeline; across three seeds, its deeper exit ties the separate gate. A human evaluation (1,144 ratings over 32 dialogues) supports the memory effect and rates the single model level with the matched 1B pipeline and below the 8B one. Graded metrics and LlamaPIE's native judge track human ratings more closely than exact match, which we report as a strict lower bound.
BibTeX
@misc{talukder2026when,
  author = {A. J. A. Jubayer Talukder and Mohammad Ariful Haque},
  title = {{When to Assist and What to Suggest: Joint Gating and Memory-Grounded Hint Generation in One Compact Decoder}},
  year = {2026},
  eprint = {2610.40611},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40611},
}
Figure