TL;DR
RSIV-Harness is a novel framework that enables conversational text-to-visualization agents to improve themselves by evolving their "harness"—the prompts, tools, and logic surrounding a frozen Large Language Model (LLM)—based on their own failure traces. Alongside this method, the authors introduce ODAC-Vis, a rigorous new benchmark that challenges agents to discover relevant data across hundreds of databases, resolve ambiguous user requests, and refine charts through multi-turn dialogue.
What Problem Does This Paper Address?
Creating a data visualization from a natural language request is rarely a one-shot process. In real-world enterprise settings, users often don't know which database contains their data, their requests are frequently ambiguous (e.g., "show sales trend" could mean gross or net sales), and they often want to iterate on a chart once they see it.
Current agents typically rely on a static "harness"—a fixed set of system prompts, tool interfaces, and control policies. When these agents fail, they keep failing in the same way because the harness never changes. While one could retrain the underlying LLM, this is computationally expensive and often unnecessary. The authors argue that the bottleneck isn't just the model's weights, but the rigid environment (the harness) in which the model operates.
Methods
The authors propose a bi-level recursive self-improvement (RSI) architecture:
- Inner Task Loop: This is the "acting" layer. It uses a ReAct (Reason + Act) pattern where the agent analyzes a request, uses tools to discover data, asks clarifying questions if needed, and eventually generates visualization code (Python/Plotly).
- Outer Self-Improvement Loop: This is the "learning" layer. It examines traces of the agent's failures across many tasks. It identifies recurring failure patterns and autonomously proposes revisions to the harness (e.g., updating a prompt to be more specific or adding a new "skill" to the middleware).
To ensure this "evolution" doesn't break things that were already working, they implement a four-layer stability framework:
- Anchor Checks: The new harness must maintain or improve performance on a fixed validation set.
- Memory Checks: It uses a "regression memory" to ensure the agent doesn't lose the ability to solve tasks it previously mastered.
- Refine/Restart Rules: If a proposed change fails these checks, the system reverts to the last "champion" harness.
Datasets
The paper introduces ODAC-Vis (Open-Domain, Ambiguous, and Conversational text-to-visualization), a benchmark consisting of 640 tasks across 206 SQLite databases.
Unlike previous benchmarks that give the agent the correct table upfront, ODAC-Vis requires:
- Open-Domain Discovery: Finding the right database among hundreds.
- Ambiguity Resolution: Handling eight types of ambiguity (e.g., column selection, aggregation granularity, chart type).
- Conversational Refinement: Supporting "drilldown" (deepening analysis) or "iteration" (changing the chart) scenarios.
Key Insights and Ideas
- Harness Evolution vs. Weight Tuning: You can achieve massive performance gains by optimizing the instructions and tools given to an LLM rather than the LLM itself. This keeps the core model "frozen" and efficient.
- Stability is Non-Negotiable: Open-ended self-improvement is risky. The authors' formal stability guarantees (like anchor monotonicity) are crucial for preventing the agent from "drifting" into worse behaviors.
- Cross-Backbone Transfer: Interestingly, a harness evolved using one model (e.g., DeepSeek) can be successfully transferred to a different model (e.g., GPT-6 Luna), suggesting that the learned "skills" and "prompts" capture general best practices for the task.
Key Results
The researchers found that just a single round of recursive self-improvement led to dramatic gains:
- Executability: Jumped by +28.02 points compared to a static seed harness.
- Chart Structure: Improved by +34.01 points, meaning the final visualizations were much closer to the user's actual intent.
- Stance Accuracy: The agent became significantly better at knowing when to ask a clarifying question versus when to just write the code.
In case studies, RSIV-Harness was the only method capable of correctly joining multiple databases while respecting specific user constraints like excluding certain data points and encoding variables with specific colors.
Limitations and Future Work
- Single Round Evolution: The experiments primarily focused on one round of improvement. The long-term dynamics of many successive rounds (potential for "overfitting" the harness to the training panel) remain to be explored.
- Computational Overhead: While cheaper than retraining an LLM, the outer loop requires running many tasks to generate failure traces, which can be time-consuming.
- Dependency on Proposal Model: The quality of the harness revision depends on the LLM used in the outer loop to analyze failures. If the "analyst" model is weak, the proposed improvements may be suboptimal.
Future work could involve extending this to even more complex data types (like NoSQL or unstructured data lakes) and exploring multi-agent harness evolution where different agents specialize in different parts of the visualization pipeline.
Summary updated