Joint Agent Memory and Exploration Learning via Novelty Signals

Shizuo Tian, Xiaohong Weng, Rui Kong, Yuxuan Chen, Guohong Liu, Yuebing Song, Jiacheng Liu, Yuchen Li, Dawei Yin, Ting Cao, Yunxin Liu, Yuanchun LiarXiv 2026 · arXiv

TL;DR

JAMEL (Joint Agent Memory and Exploration Learning) introduces a framework that jointly trains an LLM agent's latent memory and exploration policy using annotation-free, deterministic novelty signals like code coverage. This approach allows the agent to compress long interaction histories into compact memory tokens, achieving the exploration depth of full-history closed-source models while drastically reducing token consumption.

What Problem Does This Paper Address?

For autonomous agents operating in open-ended environments (like web browsers or operating systems), effective exploration is crucial because external rewards are usually absent or extremely sparse. To explore efficiently without running in circles, an agent needs a robust memory of where it has already been.

While retaining the entire raw interaction history (text prompts and screenshots) works well, it quickly becomes computationally prohibitive and hits context window limits as trajectories grow. "Latent memory" offers a solution by compressing histories into compact vector embeddings. However, training these latent memory modules is notoriously difficult because we lack step-by-step, human-readable supervisory signals showing exactly what the memory should encode at any given moment.

Methods

The authors introduce JAMEL, a framework built on the insight that memory and exploration form a mutually dependent loop: sustained exploration requires memory to avoid repeated behaviors, while novelty-seeking interactions provide the natural supervision needed to train that memory.

Figure 1: Architecture of JAMEL.

The JAMEL framework consists of three main components:

  1. History Compression Architecture: A frozen vision-language model (Qwen3-VL-2B-Instruct) acts as the compressor. At each step, it condenses the current observation-action pair into a single latent memory token (the hidden state of the EOS token).
  2. Policy Integration: A learned linear aligner projects these accumulated memory tokens into the embedding space of the main policy model (Qwen2.5-VL-7B-Instruct), where they are prepended as a soft prefix to guide the next action.
  3. Novelty-Driven Rejection Fine-Tuning (RFT): To train the system without manual annotations, JAMEL leverages code coverage (lines, branches, statements, and functions executed in the application backend) as a deterministic proxy for novelty. If an action increases cumulative code coverage, it receives a binary reward ($r_t = 1$). The agent is trained via RFT on successful exploration prefixes to maximize the likelihood of discovering new states.

Datasets

The framework was trained and evaluated on ScaleWoB, a benchmark consisting of real-world web applications (including e-commerce, social media, travel, and productivity tools).

  • Training: 86 web applications were used to collect a dataset of 24k training samples via an exploratory LLM pipeline.
  • Evaluation: 10 held-out, completely unseen applications were used to test the agent's generalization capabilities over 50-step sessions.

Key Insights and Ideas

  • Code Coverage as an Intrinsic Reward: Instead of relying on fragile LLM self-judgments or complex world-model prediction errors, software code coverage provides a persistent, un-gameable metric for behavioral novelty. Once a code path is triggered, it can never yield a novelty reward again in that session, preventing the agent from getting stuck in repetitive loops.
  • The Natural Curriculum: Novelty rewards are naturally dense at the beginning of a session when shallow, easy-to-trigger interactions are abundant. As those are exhausted, rewards become progressively sparser, forcing the policy and memory to co-evolve and discover deeper, multi-step navigation paths without explicit curriculum design.

Key Results

Evaluated on the 10 unseen test applications, JAMEL-9B demonstrated a powerful balance of exploration depth and computational efficiency:

  • Exploration Performance: JAMEL achieved an average cumulative coverage reward of 20.7, significantly outperforming open-source baselines like MAI-UI-8B (8.4) and Mobile-Agent-v3.5 (5.9).
  • Rivaling Closed-Source Models: It effectively matched the performance of closed-source models utilizing full, unpruned interaction histories, sitting right between Gemini 3.1 Flash-Lite ReAct-text (19.9) and ReAct-vision (20.9).
Figure 2: Reward accumulation on test apps. Average cumulative coverage reward across 10 test apps over a 50-step session. Shaded bands denote standard error across apps.
  • Token Efficiency: Because it compresses history into compact latent tokens rather than passing raw text and screenshots at every step, JAMEL consumed only 2,122 tokens per step. This is roughly 2.8x fewer tokens than competing open-source baselines and up to 21.9x fewer tokens than full-history ReAct-vision.

Limitations and Future Work

Despite its strong performance, JAMEL remains susceptible to "entrapment" in exceptionally dense or poorly structured user interfaces. For instance, in complex applications like Pinduoduo, the agent frequently encountered prolonged plateaus. This typically happens due to persistent modal overlays (pop-ups), where the agent wastes steps trying to click visually apparent background elements that are functionally blocked by the active foreground window.

For future work, the authors suggest exploring the scaling laws of this architecture (larger models and longer trajectories), integrating explicit Reinforcement Learning (RL) strategies, and transitioning to an "explore-then-execute" paradigm where the accumulated latent memory is directly used to complete specific downstream user tasks.

Summary updated

Abstract

In open-ended environments, exploration is fundamental for autonomous agents, yet current language model agents struggle with this. Effective exploration requires memory, but retaining raw interaction histories is computationally expensive over long trajectories. While latent memory offers a solution to compress interaction histories, its training lacks reliable supervisory signals. We introduce \textbf{J}oint \textbf{A}gent \textbf{M}emory and \textbf{E}xploration \textbf{L}earning (\textbf{JAMEL}), a framework that trains agentic memory and exploration policy together through novelty-driven interaction. We observe that memory and exploration form a mutually dependent loop: sustained exploration requires memory to distinguish exhausted behaviors from unseen ones, while novelty-seeking interaction provides the supervision needed to make memory useful for future exploration. By utilizing deterministic and persistent novelty signals such as code coverage in the GUI domain, we provide natural, annotation-free supervision for the memory module. Empirical evaluations demonstrate that \ours successfully generalizes to unseen environments. Its exploration capability outperforms open-weight baselines and rivals the exploration depth of a closed-source model while reducing token consumption. Our code and model are open-sourced at https://github.com/MobileLLM/JAMEL.
BibTeX
@misc{tian2026joint,
  author = {Shizuo Tian and Xiaohong Weng and Rui Kong and Yuxuan Chen and Guohong Liu and Yuebing Song and Jiacheng Liu and Yuchen Li and Dawei Yin and Ting Cao and Yunxin Liu and Yuanchun Li},
  title = {{Joint Agent Memory and Exploration Learning via Novelty Signals}},
  year = {2026},
  eprint = {2606.01528},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.01528},
}
Figure