WISE: An Agentic Framework for Whole-page Multimodal Search Experience Attribution

Wenyi Xu, Renzhe Zhou, Songyang Li, Feiran Zhu, Ruoqi Zhao, Chenglei Dai, Xuanping Li, Kaiqiao Zhan, Jingwei Zhuo

TL;DR

Researchers at Kuaishou Technology have introduced WISE, an agentic framework designed to diagnose not just if a search result is bad, but why it failed across a complex algorithmic pipeline. By combining a novel trajectory repair method (ReTrace) and an automated optimization loop (AHPO), they achieved a 99.55% reduction in diagnostic costs and significantly improved production search relevance.

What Problem Does This Paper Address?

Modern search engines are "black boxes" consisting of multi-stage pipelines: retrieval, ranking, and re-ranking. Even when a system is operationally healthy (no crashes, low latency), it can still return poor-quality results—a "silent failure."

Traditional monitoring focuses on reliability, while quality evaluation usually stops at identifying a "bad case." The authors argue that we need System Quality Diagnosis (SQD): a way to automatically attribute user-facing quality issues (like a species mismatch in a "yellow catfish" query) to specific internal components (like an overestimated relevance score). This is challenging because it requires both semantic understanding of multimodal content (video, text, ASR) and grounded reasoning over internal system traces.

Methods

The authors propose WISE (Whole-page Multimodal Search Experience Attribution), an agentic framework that treats diagnosis as a multi-step reasoning task.

Figure 2: Overview of the WISE framework. The bottom row shows the staged analysis process, while the agent harness coordinates model inference, context construction, evidence access, state management, and validation throughout execution.

Key technical innovations include:

  • Agentic Formulation: The system uses an LLM Policy (the "brain" for reasoning) and an Agent Harness (the "body" that manages tools, state, and evidence access).
  • ReTrace (Trajectory Repair): Expert labels tell you the final answer but not the reasoning steps. ReTrace uses a teacher model to identify the earliest error in an agent's execution and "repairs" the trajectory suffix to match the expert's conclusion. This creates high-quality, evidence-consistent data for Supervised Fine-Tuning (SFT).
  • AHPO (Adaptive Harness-Policy Optimization): A failure-driven optimization loop. When the agent fails, a "Failure Analyzer" determines if the error was a Policy Error (the LLM was "dumb") or a Harness Limitation (the LLM didn't have the right info or tool). The system then adaptively updates either the model weights or the harness configuration (prompts, schemas).

Datasets

The authors introduced WISE-BENCH, a large-scale multimodal benchmark to fill the gap in existing search datasets:

  • Scale: 12,000 real-world search cases from Kuaishou.
  • Multimodal Evidence: Includes user-visible content (text, OCR, ASR, visuals) and internal pipeline traces (recall paths, model scores, ranking positions).
  • Expert Annotations: A two-stage process where experts first identify experience issues and then inspect internal traces to attribute them to specific pipeline stages.
Figure 1: Construction and evaluation of WISE-Bench. Experts first label experience issues from request and content evidence, then inspect internal traces to annotate pipeline problems. The resulting labels support discovery, attribution, and end-to-end evaluation.

Key Insights and Ideas

  • Diagnosis is not a single-step prediction: By breaking the task into discovery, consolidation, and attribution, the agent can handle the "separation of evidence"—evaluating user experience first before looking at internal "cheating" signals.
  • Harness Optimization is as important as Policy Optimization: The paper shows that even with fixed model weights, optimizing the "body" (the harness) leads to significant gains. This suggests that many agent failures are due to poor context or tool boundaries rather than model intelligence alone.
  • Closing the Loop: The most powerful application of WISE isn't just auditing; it's using the diagnostic results to generate "hard negatives" to retrain the production relevance models, leading to measurable improvements in live search traffic.

Key Results

  • High Accuracy: WISE achieved 75.20% end-to-end exact match accuracy on the Qwen3.5-35B-A3B model.
  • Production Efficiency: In a production environment processing 120k cases daily, WISE reduced diagnostic costs from ~5,714 RMB per 1k cases (manual) to just 25.60 RMB, a 99.55% cost reduction.
  • Model Improvement: Using WISE-mined samples to retrain Kuaishou's relevance model resulted in a 4.16% advantage in human GSB (Good/Same/Bad) evaluations and a significant increase in Click-Through Rate (CTR) in online A/B tests.
Performance curves showing how AHPO (red line) consistently outperforms fixed update strategies across optimization steps.

Limitations and Future Work

  • Diversity Issues: The current benchmark has a low incidence of page-level diversity issues, so the framework's effectiveness in that specific area is less thoroughly tested.
  • Causal vs. Trace-based: The Failure Analyzer provides a trace-based account of error propagation. The authors note that while useful, this doesn't establish a full counterfactual causal effect.
  • Generalization: While focused on search, the authors suggest the SQD formulation and WISE framework could be extended to other algorithm-driven services like recommendation systems and advertising.

Summary updated

Abstract

Algorithm-driven services, including search, recommendation, and advertising, rely on multi-stage model pipelines. While research prioritizes model optimization, post-deployment monitoring focuses on operational reliability. Yet operationally healthy systems may still produce poor-quality results, exposing a gap in existing evaluation and monitoring practices. However, a systematic problem formulation and benchmark for this task remain lacking. To bridge this gap, we formulate System Quality Diagnosis (SQD) for algorithm-driven services, a new task that extends quality evaluation beyond identifying poor outcomes to attributing their causes across system components. Focusing on industrial search, we propose WISE, an agentic framework for end-to-end quality diagnosis that discovers user-facing issues and traces their origins to underlying algorithmic components. To improve diagnostic reliability, we introduce ReTrace for constructing evidence-consistent trajectory supervision and Adaptive Harness-Policy Optimization (AHPO) for failure-guided optimization of the agent harness and model policy. To enable systematic evaluation, we construct WISE-BENCH, a real-world multimodal search benchmark that combines user-facing evidence, internal pipeline traces, and expert diagnosis annotations. Extensive experiments show consistent gains across model scales, with WISE achieving 75.20% end-to-end accuracy on Qwen3.5-35B-A3B. In production, WISE supports quality auditing, online quality assurance, and model optimization, reducing diagnosis costs and improving downstream relevance models.
BibTeX
@misc{xu2026wise,
  author = {Wenyi Xu and Renzhe Zhou and Songyang Li and Feiran Zhu and Ruoqi Zhao and Chenglei Dai and Xuanping Li and Kaiqiao Zhan and Jingwei Zhuo},
  title = {{WISE: An Agentic Framework for Whole-page Multimodal Search Experience Attribution}},
  year = {2026},
  eprint = {2611.06777},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06777},
}
Figure