TL;DR
Researchers at Kuaishou Technology have introduced WISE, an agentic framework designed to diagnose not just if a search result is bad, but why it failed across a complex algorithmic pipeline. By combining a novel trajectory repair method (ReTrace) and an automated optimization loop (AHPO), they achieved a 99.55% reduction in diagnostic costs and significantly improved production search relevance.
What Problem Does This Paper Address?
Modern search engines are "black boxes" consisting of multi-stage pipelines: retrieval, ranking, and re-ranking. Even when a system is operationally healthy (no crashes, low latency), it can still return poor-quality results—a "silent failure."
Traditional monitoring focuses on reliability, while quality evaluation usually stops at identifying a "bad case." The authors argue that we need System Quality Diagnosis (SQD): a way to automatically attribute user-facing quality issues (like a species mismatch in a "yellow catfish" query) to specific internal components (like an overestimated relevance score). This is challenging because it requires both semantic understanding of multimodal content (video, text, ASR) and grounded reasoning over internal system traces.
Methods
The authors propose WISE (Whole-page Multimodal Search Experience Attribution), an agentic framework that treats diagnosis as a multi-step reasoning task.
Key technical innovations include:
- Agentic Formulation: The system uses an LLM Policy (the "brain" for reasoning) and an Agent Harness (the "body" that manages tools, state, and evidence access).
- ReTrace (Trajectory Repair): Expert labels tell you the final answer but not the reasoning steps. ReTrace uses a teacher model to identify the earliest error in an agent's execution and "repairs" the trajectory suffix to match the expert's conclusion. This creates high-quality, evidence-consistent data for Supervised Fine-Tuning (SFT).
- AHPO (Adaptive Harness-Policy Optimization): A failure-driven optimization loop. When the agent fails, a "Failure Analyzer" determines if the error was a Policy Error (the LLM was "dumb") or a Harness Limitation (the LLM didn't have the right info or tool). The system then adaptively updates either the model weights or the harness configuration (prompts, schemas).
Datasets
The authors introduced WISE-BENCH, a large-scale multimodal benchmark to fill the gap in existing search datasets:
- Scale: 12,000 real-world search cases from Kuaishou.
- Multimodal Evidence: Includes user-visible content (text, OCR, ASR, visuals) and internal pipeline traces (recall paths, model scores, ranking positions).
- Expert Annotations: A two-stage process where experts first identify experience issues and then inspect internal traces to attribute them to specific pipeline stages.
Key Insights and Ideas
- Diagnosis is not a single-step prediction: By breaking the task into discovery, consolidation, and attribution, the agent can handle the "separation of evidence"—evaluating user experience first before looking at internal "cheating" signals.
- Harness Optimization is as important as Policy Optimization: The paper shows that even with fixed model weights, optimizing the "body" (the harness) leads to significant gains. This suggests that many agent failures are due to poor context or tool boundaries rather than model intelligence alone.
- Closing the Loop: The most powerful application of WISE isn't just auditing; it's using the diagnostic results to generate "hard negatives" to retrain the production relevance models, leading to measurable improvements in live search traffic.
Key Results
- High Accuracy: WISE achieved 75.20% end-to-end exact match accuracy on the Qwen3.5-35B-A3B model.
- Production Efficiency: In a production environment processing 120k cases daily, WISE reduced diagnostic costs from ~5,714 RMB per 1k cases (manual) to just 25.60 RMB, a 99.55% cost reduction.
- Model Improvement: Using WISE-mined samples to retrain Kuaishou's relevance model resulted in a 4.16% advantage in human GSB (Good/Same/Bad) evaluations and a significant increase in Click-Through Rate (CTR) in online A/B tests.
Limitations and Future Work
- Diversity Issues: The current benchmark has a low incidence of page-level diversity issues, so the framework's effectiveness in that specific area is less thoroughly tested.
- Causal vs. Trace-based: The Failure Analyzer provides a trace-based account of error propagation. The authors note that while useful, this doesn't establish a full counterfactual causal effect.
- Generalization: While focused on search, the authors suggest the SQD formulation and WISE framework could be extended to other algorithm-driven services like recommendation systems and advertising.
Summary updated