Caesar: Deep Agentic Web Exploration for Creative Answer Synthesis

Jason Liang, Elliot Meyerson, Risto MiikkulainenarXiv 2026 · arXiv

TL;DR

Caesar is a novel agentic architecture designed to move AI web search beyond flat information retrieval and derivative summaries toward true creative synthesis. By combining deep web traversal to build a dynamic knowledge graph with an adversarial refinement loop, Caesar uncovers non-obvious connections and generates answers that are significantly more novel, useful, and surprising than current state-of-the-art deep research agents.

What Problem Does This Paper Address?

While Large Language Models (LLMs) excel at following instructions and summarizing text, current web-searching agents suffer from "navigational amnesia." Architectures based on standard Retrieval-Augmented Generation (RAG) or linear ReAct loops treat the internet as a flat sequence of disconnected documents. They optimize heavily for precision—finding the most probable answer to a known question—but fail at open-ended inquiry.

When faced with complex challenges requiring "combinatorial creativity" (bridging disparate conceptual islands), standard agents default to shallow, mean-seeking summaries. Caesar addresses this gap by treating web exploration as a stateful graph traversal problem, mapping out the web's topology to discover diverse, orthogonal perspectives before synthesizing an answer.

Methods

Caesar operates through a two-phase cognitive cycle:

  1. Phase 1: Deep Web Exploration: Instead of linear scraping, Caesar utilizes a recursive Perceive-Think-Act loop to build a topological knowledge graph ($G$) alongside a semantic knowledge base ($KB$).
  2. Domain-Specific Role Adaptation: Caesar dynamically rewrites its own system prompt to adopt a persona tailored to the creative constraints of the task.
  3. Graph-Augmented Insight Generation: The agent conditions its information extraction on the local graph neighborhood, allowing it to identify how new content builds upon or contradicts past discoveries.
  4. Knowledge-Guided Exploration: A dynamic decision policy uses episodic memory to detect navigation stagnation, autonomously switching between depth-first expansion and strategic backtracking.
  5. Phase 2: Adversarial Artifact Synthesis: Rather than summarizing the gathered information in a single shot, Caesar emulates a human drafting and critiquing process.
  6. Recursive Insight Discovery: The agent runs a chain of inquiries to target ambiguities and gaps in its knowledge.
  7. Adversarial Challenge Refinement: Between drafting rounds, Caesar actively analyzes its current draft for narrative weaknesses and formulates new, orthogonal challenges to push itself out of generic LLM "basins of attraction."
  8. Generative Draft Merging: Finally, an LLM selectively integrates the most relevant insights from the entire draft history into a single, cohesive narrative.
Figure 1: Visualization of the Caesar architecture. (Left) Phase 1: Deep Web Exploration. A dynamic exploration policy controls a three-stage loop (Perceive, Think, Act) to traverse the web and to build a knowledge graph/database from insights. (Right) Phase 2: Adversarial Artifact Synthesis. Insights are retrieved to synthesize an initial draft. The agent then enters a recursive cycle, critiquing the current draft to generate adversarial challenges for refinement, before consolidating all versions via a generative merge and ELI5 summary. Together, these phases transform raw web traversal into a structured reasoning process that prioritizes creativity.

Datasets

To evaluate creative synthesis, the authors constructed a dataset of five distinct challenges based on well-documented LLM failure modes:

  1. Constrained Synthesis: Invent a new human emotion and ground it in evolutionary biology.
  2. Counterfactual Reasoning: Detail how society (painting, architecture, math) would change if humans evolved with echolocation instead of color vision.
  3. Cross-Domain Synthesis: Map the formal mathematical structure of calculus onto cooking (e.g., what is a recipe's "derivative"?).
  4. Meta-Creativity: Create an objective AI creativity metric that completely excludes human judgment, novelty, usefulness, or surprise.
  5. Open-Ended Synthesis: Invent a completely original business idea that doesn't exist yet.

Evaluations were performed using a blinded LLM-as-a-Judge framework (employing Claude Sonnet 4.5, GPT-5.2, and Gemini 3 Pro) scored on a 10-point scale across three dimensions: New, Useful, and Surprising (NUS). A human evaluation study with 23 raters was also conducted to validate the automated judges.

Key Insights and Ideas

  • Topological Memory is Essential: Decoupling navigation (managed by the graph) from information retention (managed by the knowledge base) prevents the agent from looping through redundant content and helps it discover structurally connected but semantically distant concepts.
  • Creativity Thrives on Adversarial Friction: Forcing the agent to generate adversarial counter-questions against its own drafts successfully breaks the default, highly predictable pattern-matching behavior of standard LLMs.
  • Substance Over Style: Caesar’s performance advantage actually widens or remains dominant under strict length constraints (e.g., a 450-word limit). This proves its success stems from high-density semantic content rather than verbose, flowery prose.

Key Results

  • Strict Dominance: Caesar achieved a 13% to 23% overall improvement over state-of-the-art deep research agents (such as Gemini 3 Deep Research and Claude Sonnet 4.5 Research Mode) across all output formats.
  • NUS Scoring: In unconstrained full answers, Caesar achieved a total score of 26.96 out of 30, significantly outperforming the closest runner-up, Gemini 3 Deep (23.78).
LLM-as-a-Judge Results Across Output Constraints
  • Compute-Controlled Parity: When restricted to a strict budget ($5 per challenge using GPT-5-mini), Caesar still scored 26.16, outperforming deep baselines that utilize significantly more compute.
  • Human Validation: Human raters preferred Caesar's creative outputs in 56.25% of pairwise A/B matchups against the strongest deep research baseline.

Limitations and Future Work

  • High Computational Overhead: Caesar's deep exploration phase is expensive, averaging $40 to $60 per 1000-step experiment due to heavy token ingestion and link processing. It is designed for asynchronous research rather than real-time interaction.
  • The SEO Trap: If the initial search bootstrapping seeds the graph within a cluster of low-quality, search-optimized boilerplate content, the agent can get trapped in recursive loops of low-value information.
  • Diminishing Returns on Pure Speculation: On purely speculative tasks that rely on parametric invention rather than retrievable evidence (like open-ended business ideation), Caesar's lead narrows.
  • Future Directions: The authors plan to extend Caesar to collaborative multi-agent swarms, apply the architecture to non-QA domains like program synthesis (e.g., the ARC-AGI benchmark), and introduce adaptive mechanisms to dynamically scale exploration depth based on task complexity.

Summary updated

Abstract

To advance from passive retrieval to creative discovery of new ideas, autonomous agents must be capable of deep, associative synthesis. However, current agentic frameworks prioritize convergent search, often resulting in derivative summaries that lack creativity. Caesar is an agentic architecture designed to bridge the gap between information gathering and synthesis of new insights. Unlike existing agents that treat the web as a flat sequence of disconnected documents, Caesar performs a deep web traversal to construct a dynamic knowledge graph. This graph then serves as a navigational scaffold, guiding the agent to diverse, non-obvious information that flat retrieval would never encounter. Caesar thus consists of two components: (1) exploration driven by a dynamic context-aware policy that maximizes information coverage across the web's topological structure, and (2) synthesis through adversarial refinement that actively seeks novel perspectives rather than confirming established priors. Caesar demonstrates the ability to generate artifacts and answers characterized by high novelty and structural coherence, achieving 13% to 23% improvement over state-of-the-art deep research agents in creative synthesis challenges, with strong dominance across all output formats.
BibTeX
@misc{liang2026caesar,
  author = {Jason Liang and Elliot Meyerson and Risto Miikkulainen},
  title = {{Caesar: Deep Agentic Web Exploration for Creative Answer Synthesis}},
  year = {2026},
  eprint = {2604.20855},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.20855},
}

Loading related content...

Figure