TL;DR
This paper introduces a workflow that uses LLM agents guided by Knowledge Graphs to "recover" deterministic business rules from historical enterprise data. Instead of training opaque statistical models, the system generates transparent, high-speed "cascade lookups" that outperform Graph Neural Networks (GNNs) on enterprise tasks while providing built-in auditability and drift diagnostics.
What Problem Does This Paper Address?
Enterprise Resource Planning (ERP) systems like SAP run on "Condition Techniques"—complex sets of rules that determine fields like payment terms, shipping points, and sales ownership based on keys like customer ID or material type. However, these rules are often manually maintained, leading to a "configuration gap" where rules become stale or incomplete.
When rules fail, human operators must fill in the blanks manually. This is slow, expensive, and error-prone; a wrong shipping point can route goods to the wrong warehouse, while incorrect payment terms lead to billing disputes. Traditional machine learning approaches (like XGBoost or GNNs) treat this as a statistical pattern recognition problem. However, enterprise data is often governed by configured determinism—explicit rules rather than fuzzy correlations. Statistical models often struggle with the high cardinality of these fields (e.g., hundreds of possible sales teams) and lack the deterministic "audit trail" required for enterprise compliance.
Methods
The authors propose a system called CascadER, which moves the heavy lifting of LLMs to an offline "discovery" phase.
- Rule Discovery (Explore): An LLM agent (e.g., Claude Opus) acts as a data analyst. It reads a Knowledge Graph (KG) containing SAP metadata to understand which fields should logically influence others.
- SQL Profiling: The agent follows a "Propose, Profile, Score, Refine" loop. It proposes candidate "business keys" (e.g., "Does
Customer+DocTypedeterminePaymentTerms?"), generates SQL queries to test these against historical data, and validates them based on accuracy and coverage. - Cascade Codification (Build): The discovered rules are compiled into a JSON Cascade. This is a hierarchy of lookups ranging from specific (Customer + Material) to general fallbacks (Global Mode).
- Runtime Deployment: At inference time, the system performs a simple key lookup in the JSON file. There is no LLM or GPU required at runtime, ensuring sub-millisecond latency and 100% reproducibility.
- Drift Monitoring: The system continuously compares new transactions against the cascade. If the accuracy of a rule drops, it triggers a "Governance Alert," helping admins identify exactly where business processes have changed (e.g., a sales team reorganization).
Datasets
- SALT-KG: A benchmark containing 2.3 million historical order items from a real-world SAP S/4HANA instance, enriched with a metadata Knowledge Graph.
- RelBench v2 (Generalization Study): To test the limits of the approach, the authors used
rel-arxiv(citation data) andrel-stack(StackOverflow badges). These datasets lack the strict "business rule" structure of ERP systems.
Key Insights and Ideas
- Domain Knowledge > Model Complexity: In enterprise settings, knowing which columns to look at (provided by the KG) is more valuable than using a more sophisticated architecture. The KG helps prune a combinatorially large search space of possible column combinations.
- The "Configured Determinism" Gap: GNNs fail on enterprise data because they try to smooth patterns across "neighborhoods." In a rule-based system, two customers might be "similar" in every way but have different payment terms due to a specific contract—a distinction a lookup table preserves but an embedding might blur.
- Drift as a Diagnostic: By measuring "cross-drift" (how often field A changes when field B changes), the system can map causal directions in business logic. For example, the authors found that when a
Plantassignment changes, theShippingPointchanges 95% of the time, but the reverse is not true.
Key Results
The cascade method outperformed all baselines, including GNNs and gradient-boosted trees, on all 8 SALT-KG fields.
- Accuracy Gains: On the
SalesGroupfield (which has 589 possible values), a GNN achieved only 15.8% accuracy, while the Cascade reached 70.0%. - Operational Impact: For
CustomerPaymentTerms, the cascade reduced the manual correction rate from 62.5% (GNN baseline) to just 17.1%. - Efficiency: The entire prediction engine for 8 fields fits in a 12 MB JSON file, compared to ~200 MB for model weights, and requires zero GPU resources for serving.
Limitations and Future Work
- Requirement for Rules: The generalization study showed that cascades underperform GNNs on non-enterprise data (like StackOverflow badges) where targets are determined by temporal thresholds or fuzzy social patterns rather than stable rules.
- Cold-Start Problem: Like all history-based methods, cascades struggle with new entities (e.g., a brand-new customer) that have no historical transactions to derive rules from.
- Future Directions: The authors suggest "temporal cascades" that can handle threshold-based rules and hybrid "Cascade-GNN" architectures that use rule-based lookups as features for a neural model to get the best of both worlds.
Summary updated