Memory-Augmented Agent Training for Business Document Understanding

Jiale Liu, Yifan Zeng, Malte Højmark-Bertelsen, Marie Normann Gadeberg, Huazheng Wang, Qingyun WuarXiv 2024 · OpenReview

Abstract

Traditional enterprises face significant challenges in processing business documents, where tasks like extracting transport references from invoices remain largely manual despite their crucial role in logistics operations. While Large Language Models offer potential automation, their direct application to specialized business domains often yields unsatisfactory results. We introduce Matrix (Memory-Augmented agent Training through Reasoning and Iterative eXploration), a novel paradigm that enables LLM agents to progressively build domain expertise through experience-driven memory refinement and iterative learning. To validate this approach, we collaborate with one of the world's largest logistics companies to create a dataset of Universal Business Language format invoice documents, focusing on the task of transport reference extraction. Experiments demonstrate that Matrix outperforms prompting a single LLM by 30.3% and a vanilla LLM agent by 35.2%. We further analyze the metrics of the optimized systems and observe that the agent system requires fewer API calls, incurs lower costs, and can analyze longer documents on average. Our methods establish a new approach to transform general-purpose LLMs into specialized business tools through systematic memory enhancement in document processing tasks.

Introduction

In global logistics, processing invoices is a high-stakes, manual nightmare. Extracting specific identifiers—like transport reference numbers—from complex, structured formats like Universal Business Language (UBL) is notoriously error-prone. While Large Language Models (LLMs) excel at general reasoning, they often struggle in specialized domains because they lack familiarity with specific structural quirks and business document patterns. Existing solutions typically rely on manual prompt engineering, which is difficult to scale, or standard agentic workflows that lack mechanisms to learn from past mistakes. We address this gap by proposing a system that allows LLM agents to systematically refine domain expertise through experience, much like a human trainee.

The Matrix Framework

The core of our approach is the Matrix (Memory-Augmented agent Training through Reasoning and Iterative eXploration) framework. Matrix operates as a closed-loop system designed to distill reasoning "trajectories" into a long-term memory module.

The Matrix framework consists of an Inference Stage and a Training Stage, where experience is iteratively distilled into a memory module.

The framework follows a four-stage iterative process:

  1. Trajectory Sampling: The agent attempts a document extraction task, recording observations, code execution, and actions into a "trajectory."
  2. Reflection: A post-hoc LLM evaluator (Reflector) reviews the trajectory against ground-truth data to identify exactly where the agent succeeded or failed.
  3. Optimization: A meta-optimizer (e.g., GPT-4o) takes the execution trajectory and reflection to distill generalized, actionable heuristics.
  4. Memory Update: These heuristics are stored in a long-term memory module, serving as a dynamic instruction manual for future tasks.

Unlike standard Retrieval-Augmented Generation (RAG), which might simply pull similar past examples, Matrix focuses on distilling logic. It transforms raw experience into high-level rules, such as identifying specific tags within a UBL structure when headers are ambiguous.

Experimental Setup

We collaborated with Kuehne+Nagel, a global logistics leader, to utilize real-world business invoices. The primary dataset consists of 764 valid UBL invoice documents paired with transport reference numbers. To support the research community, an anonymized subset of this data has been released, addressing the scarcity of accessible business document datasets due to privacy concerns.

Results and Performance Analysis

Matrix was evaluated against several baselines, including Chain-of-Thought (CoT) prompting, vanilla LLM agents, and the "Reflexion" self-correction method.

Comparison of success rates across different methods. Matrix significantly outperforms CoT, Two-agent, and Reflexion baselines for both gpt-4o and gpt-4o-mini.

Key findings include:

  • Superior Accuracy: Matrix outperformed CoT by 30.3% and vanilla LLM agents by 35.2%. It also surpassed the Reflexion method by 27.28%.
  • The "Mini" Advantage: In a notable result, GPT-4o-mini outperformed GPT-4o as a backbone model after Matrix optimization (55.82% vs 48.15% success rate). This suggests that high-quality distilled instructions allow smaller, faster models to punch above their weight class.
  • Efficiency Gains: Training didn't just improve accuracy; it made agents more concise. Optimized agents required 8% to 21% fewer API calls, directly translating to lower latency and operational costs.
  • Handling Long Documents: Optimized agents showed a much stronger capability in processing longer documents (6,000+ tokens), effectively finding the "needle in the haystack" where unoptimized models typically fail.

Conclusion

Matrix demonstrates that systematic memory enhancement can transform general-purpose LLMs into specialized business tools. While the system's performance is currently bounded by the optimizer's intelligence and training data quality, it offers a scalable path toward automating complex document understanding. Future work will explore coreset selection for more data-efficient learning and expanding the dataset to include more diverse edge cases, such as documents where no reference number exists.

Summary updated

Abstract

Traditional enterprises face significant challenges in processing business documents, where tasks like extracting transport references from invoices remain largely manual despite their crucial role in logistics operations. While Large Language Models offer potential automation, their direct application to specialized business domains often yields unsatisfactory results. We introduce Matrix (Memory-Augmented agent Training through Reasoning and Iterative eXploration), a novel paradigm that enables LLM agents to progressively build domain expertise through experience-driven memory refinement and iterative learning. To validate this approach, we collaborate with one of the world's largest logistics companies to create a dataset of Universal Business Language format invoice documents, focusing on the task of transport reference extraction. Experiments demonstrate that Matrix outperforms prompting a single LLM by 30.3%, vanilla LLM agent by 35.2%. We further analyze the metrics of the optimized systems and observe that the agent system requires less API calls, fewer costs and can analyze longer documents on average. Our methods establish a new approach to transform general-purpose LLMs into specialized business tools through systematic memory enhancement in document processing tasks.
BibTeX
@misc{liu2024memory,
  author = {Jiale Liu and Yifan Zeng and Malte Højmark-Bertelsen and Marie Normann Gadeberg and Huazheng Wang and Qingyun Wu},
  title = {{Memory-Augmented Agent Training for Business Document Understanding}},
  year = {2024},
  eprint = {2412.15274},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.15274},
}
Figure