TL;DR
KDASO is a novel framework that automates the creation, refinement, and selection of "skills"—executable reasoning rules—for high-stakes liability adjudication tasks. By combining data-driven induction with expert-aligned refinement, it enables Large Language Models (LLMs) to handle evolving real-world scenarios (like new types of ride-hailing fraud) with higher precision, better interpretability, and significantly reduced "skill omission" in long contexts.
What Problem Does This Paper Address?
Liability adjudication involves analyzing event descriptions (e.g., a ride-hailing dispute or a legal case) against specific rules to determine who is at fault. While LLMs are promising for this, the authors identify three "intrinsic tensions" in current skill-based methods:
- Distribution Shift: Real-world scenarios evolve faster than human experts can write rules. For example, new "virtual displacement" fraud in ride-hailing requires new detection logic that static libraries lack.
- Semantic Execution Gap: Even if a rule is logically sound, an LLM might interpret the natural language inconsistently. It might reach the right conclusion through flawed reasoning, or fail because the rule's phrasing is ambiguous.
- Context Budget Constraints: As a skill library grows, you can't just cram every rule into the LLM's prompt. This leads to "skill omission," where the model ignores relevant rules, or "attentional interference," where irrelevant rules distract it.
Methods
The KDASO (Knowledge-Data Dual-Driven Automated Skill Optimization) framework consists of three synergistic modules:
- Neighborhood-Conditioned Skill Induction: To avoid overfitting to a single case or being overwhelmed by the entire dataset, KDASO identifies a "neighborhood" of semantically similar recent cases. It then induces new skills from this local manifold, ensuring the rules are both representative and computationally tractable.
- Execution-Aligned Skill Refinement: This module bridges the semantic gap. It analyzes discrepancies between the LLM’s reasoning trajectories and expert reasoning paths. If the LLM's logic diverges (even if the final label is correct), the system automatically restructures the skill description using "execution scaffolding" to stabilize performance.
- Context-Budgeted Skill Routing: Instead of providing all skills, KDASO trains lightweight XGBoost classifiers for each skill. At test time, it dynamically routes only the most relevant subset of skills to the LLM, ensuring the prompt stays within the context budget while maintaining high recall.
Datasets
The authors evaluated KDASO across three diverse domains:
- Ride-hailing Adjudication: Five closed-source datasets from DiDi (CA, CR, CPC, CDC, FD) covering disputes like detours and cancellations.
- Legal Liability Determination: Tasks from LawBench (Issue Topic Identification, Charge Prediction) and LegalBench (Unfair Terms of Service, etc.).
- Logical Reasoning Judgment: Standard benchmarks ReClor and AR-LSAT.
Key Insights and Ideas
- Description Quality over Rule Presence: Simply having a rule isn't enough. Figure 1 shows that "following experts"—providing detailed, step-by-step verification logic—significantly boosts precision compared to plain text or basic knowledge-base entries.
- The Scalability Paradox: Indiscriminately adding skills to a prompt eventually hurts performance. The authors found a strong positive correlation ($R^2 = 0.82$) between the number of input skills and the "skill omission ratio" (Figure 2).
- Skill Transferability: Skills generated for one scenario (e.g., driver appeals) can be effectively transferred to similar scenarios (e.g., passenger cancellations), yielding an 8.96% performance boost without further adaptation.
- Continuous Evolution: By using a "rolling update" mechanism, KDASO maintains stable performance gains over time, effectively assimilating new data distributions as they emerge.
Key Results
KDASO consistently outperformed both general-purpose LLMs and domain-specific baselines:
- Ride-hailing: On the "Cancel-Reservation" (CR) task, KDASO achieved a 16% absolute gain in O-Precision over the strongest baseline, a critical metric for maintaining user trust in industrial settings.
- Legal: It surpassed proprietary models like GPT-4o on several LawBench and LegalBench tasks, demonstrating that explicit skill optimization can beat raw parameter scale.
- Logic: KDASO currently ranks first on the official ReClor leaderboard, achieving a score gain of 3.5 over previous logic-specific models.
Limitations and Future Work
The authors acknowledge that the current induction process relies on embedding-based neighborhoods, which might miss "boundary cases" that are semantically distant but legally critical.
Future work aims to:
- Integrate more rigorous symbolic verification layers (like NL2FOL) to further reduce logical contradictions and hallucinations.
- Test the framework's robustness under substantial domain shifts, such as cross-regulatory or cross-lingual adjudication.
- Reduce the reliance on textual alignment with human experts by moving toward more formal logical representations.
Summary updated