KARMA: Training-Free Knowledge-Atomic Harness Evolution for Industrial Advertising Agents

Xinhua Miao, Yuanzhen Rong, Weiru Zhang, Xiaoyi Zeng

TL;DR

KARMA is a training-free framework that improves LLM-based advertising agents by evolving their decision-making knowledge through structured, "atomic" edits. By keeping model weights and core code frozen while iteratively refining a modular knowledge base, it achieves a ~30% performance boost across diverse e-commerce domains.

What Problem Does This Paper Address?

In industrial settings like e-commerce advertising, agents must constantly adapt to new products, shifting market trends, and updated business policies. Traditionally, developers have two main ways to update these agents:

  1. Fine-tuning: Costly, slow, and requires massive amounts of labeled data.
  2. Prompt Engineering: Manual "whole-prompt" rewriting is difficult to scale, prone to regressions (where fixing one thing breaks another), and makes it hard to attribute exactly which change caused a specific behavior.

The authors identify a need for a system that can self-improve its decision logic in an inspectable and reversible way without the overhead of retraining or the risk of breaking the agent's underlying execution "harness" (the code that handles tool calls and workflows).

Methods

The authors introduce KARMA (Knowledge-Atomic, Regression-Gated Modular Adaptation). Instead of treating the prompt as a single block of text, KARMA externalizes knowledge into six typed modules (e.g., Experience, Constraints, Domain).

Figure 1: Motivation. Whole-prompt rewriting obscures individual changes; broad harness editing complicates validation and rollback. KARMA freezes weights and code, derives atomic knowledge edits from successes and failures, and retains only patches whose validation gain reaches 𝜃.

The framework operates in an "Outer Evolution Loop" that surrounds a frozen "Inner Decision Agent":

  • Trace Verifier: Analyzes logs to partition trajectories into successes and failures, mapping errors to specific knowledge modules.
  • Critic: A high-level LLM that compares successful and failed cases to diagnose issues (e.g., "conflicting rules" or "ambiguous wording") and proposes specific edits.
  • Evolver: A tool-constrained LLM that applies "atomic mutations" (create, update, or delete rules) to the knowledge modules.
  • Patch Validator: A crucial "gate" that runs the agent with the new knowledge on a validation set. If the performance gain doesn't meet a specific threshold ($\theta$), the change is rolled back.
Overview of the KARMA framework architecture

Datasets

The study uses a massive real-world dataset:

  • Source: Approximately 500,000 operation logs from verified merchants on a global e-commerce platform (AliExpress) from July 2026.
  • Domains: Six distinct categories, including Home & Living, Automotive, Electronics, Fashion, Beauty, and a "Long-tail" group.
  • Evaluation: 800 cases per domain, split into training batches, a validation set, and a strictly untouched hold-out set.

Key Insights and Ideas

  • Full Ingestion Matters: Most self-improving systems only look at failures. KARMA uses successes as well. This provides "negative constraints" that prevent the Evolver from proposing over-generalized rules that might break things that are already working.
  • Atomic vs. Arbitrary: By restricting the LLM to specific "atomic" edits (like changing a single rule ID) rather than letting it rewrite the whole prompt, the system maintains a clear audit trail. You can see exactly which rule was added to fix which failure.
  • Knowledge is Transferable: One of the most surprising findings was that knowledge evolved in one domain (e.g., Fashion) often improved performance in another (e.g., Electronics) without any further adaptation. This suggests the agent is learning generalizable business heuristics.
  • The "Automotive Anomaly": Not all domains are created equal. The "Automotive" domain proved to be a "weak source" of knowledge—rules learned there often harmed performance in other domains, highlighting the importance of domain-specific validation.

Key Results

KARMA was tested across 18 different domain-model settings using three LLM families (Qwen, GLM, and Kimi).

  • Performance Gain: KARMA raised the average hold-out score from 30.19 to 39.15, a relative improvement of +29.7%.
  • Ablation Study: Removing either the success trajectories or the failure trajectories from the feedback loop more than halved the performance gains, proving that "learning what to keep" is as important as "learning what to fix."
  • Stepwise Improvement: The validation gate successfully filtered out "bad ideas," resulting in a stepwise improvement curve where the agent only adopts changes that statistically prove their worth.
Validation scores showing stepwise improvements during the evolution process

Limitations and Future Work

  • Offline Evaluation: The current study measures agreement with historical logs. It does not yet measure "live" business metrics like actual revenue or click-through rates.
  • Historical Bias: Because the agent is learning from logs, it might inherit biases present in the original human or system decisions recorded in those logs.
  • Tool/Code Fixes: KARMA can only evolve knowledge. If the underlying business tools or the orchestration code itself is broken, KARMA cannot fix it.
  • Future Directions: The authors plan to explore live deployments with "human-in-the-loop" oversight and business guardrails to ensure safety in production environments.

Summary updated

Abstract

Industrial advertising agents must continually adapt their decision knowledge to changing product offerings, traffic patterns, and operating policies. We present KARMA (Knowledge-Atomic, Regression-Gated Modular Adaptation), a training-free framework that evolves six typed knowledge modules while keeping model weights and executable harness code frozen. During our evolution, a deterministic Trace Verifier maps observed errors to candidate modules, and a Critic compares successful and failed trajectories to guide a tool-constrained Evolver. The Evolver applies atomic knowledge edits, which a fixed validation gate retains only when their measured gain reaches a threshold; otherwise, the previous knowledge state is restored. We evaluate KARMA with three LLM families on cases sampled from a corpus of approximately half a million real-world operation logs across six e-commerce domains. Across 18 domain--model settings, KARMA raises the average hold-out score from 30.19 to 39.15 (+8.96 points, +29.7%). Ablations show that removing either successful or failed trajectories more than halves the average hold-out gain. Evolved knowledge also transfers across domains without further adaptation, yielding an average hold-out gain of 8.54 points across all source--target pairs. By linking retained patches to their diagnostic evidence and validation outcomes, KARMA supports inspectable, reversible knowledge adaptation for agents in Web marketplaces.
BibTeX
@misc{miao2026karma,
  author = {Xinhua Miao and Yuanzhen Rong and Weiru Zhang and Xiaoyi Zeng},
  title = {{KARMA: Training-Free Knowledge-Atomic Harness Evolution for Industrial Advertising Agents}},
  year = {2026},
  eprint = {2610.39884},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39884},
}
Figure