Introduction: The Challenge of Knowledge-Intensive Agents
Building reliable Large Language Model (LLM) agents for knowledge-intensive tasks, such as multi-hop question answering, remains a significant challenge in artificial intelligence. Current approaches often suffer from three primary limitations. First, uncontrollable logic arises when agents use "free-form" reasoning (like Chain-of-Thought), which is difficult to constrain, debug, or audit. Second, agents frequently exhibit static capabilities, struggling to adapt to the nuances of new environments or specialized knowledge bases once their initial training is complete. Finally, the reliance on sparse feedback—where agents only receive "outcome feedback" (correct/incorrect) on the final answer—makes it nearly impossible to identify which specific step in a multi-step reasoning process failed. This noisy signal hinders efficient model improvement.
To address these hurdles, the AMOR (Adaptable MOdulaR knowledge agent) framework structures LLM reasoning as a Finite State Machine (FSM), breaking complex tasks into disentangled, specialized modules. This modularity allows for precise "process feedback" on individual reasoning steps, enabling models to adapt to new domains with high data efficiency.
The AMOR Framework: FSM-Based Reasoning
AMOR formalizes the reasoning process as a Finite State Machine. Instead of a single monolithic prompt, the agent moves through a series of discrete states, each managed by a specialized module. These states include Question Decomposition, Document Retrieval, Relevance Judgment, and Answer Extraction.
By decomposing the reasoning load, each module only needs to focus on a specific, narrow input. For example, the Relevance Judgment module only evaluates whether a specific document snippet is relevant to a given sub-query, rather than attempting to hold the entire context of a complex multi-hop question. This disentanglement significantly reduces the cognitive load on the underlying LLM.
Module-Aware Mixture-of-Experts (MA-MoE)
To prevent task interference—where learning to judge relevance might degrade the model's ability to extract answers—AMOR utilizes a specialized architecture called Module-Aware Mixture-of-Experts (MA-MoE). During fine-tuning, distinct Feed-Forward Network (FFN) parameters are optimized in the final quarter of the Transformer blocks for each specific module. This allows the model to effectively switch into a "specialist" mode depending on its current state within the FSM.
Two-Stage Training and Process Feedback
The development of an AMOR agent follows a two-stage pipeline:
- Warm-up Stage: The model is initially fine-tuned on approximately 50,000 examples automatically derived from public datasets (such as 2WikiMultiHopQA and NaturalQuestions). This stage teaches the agent the fundamental mechanics of navigating the FSM and performing basic modular tasks.
- Adaptation Stage: When deployed in a new domain, the agent explores tasks and receives process feedback. Unlike outcome feedback, process feedback provides granular corrections at every junction—such as identifying a faulty sub-query or an irrelevant document. This feedback is used to further fine-tune the model using the Kahneman-Tversky Optimization (KTO) alignment algorithm, tailoring the agent to the specific requirements of the new environment.
Experimental Results
AMOR was evaluated across three diverse datasets: HotpotQA (multi-hop reasoning), PubMedQA (biomedical research), and QASPER (long-form NLP paper analysis). The results demonstrate that AMOR significantly outperforms standard agent frameworks like ReAct.
Notably, an AMOR agent built on the Llama-2-7B model achieved an Exact Match (EM) score of 45.8 on HotpotQA, surpassing a ReAct agent powered by the much larger GPT-4 (42.0). Ablation studies further confirmed that the "Process" version of AMOR consistently outperformed versions relying solely on outcome feedback or the initial warm-up stage. Furthermore, human-in-the-loop studies showed that providing expert human feedback to the modules could push performance even higher, reaching 50.8 EM on HotpotQA.
Conclusion and Future Directions
AMOR demonstrates that structured, modular reasoning combined with granular process feedback is a powerful recipe for building adaptable agents. While the current framework requires manual FSM design, it paves the way for future systems where LLMs might autonomously design their own state machines for diverse tasks. Expanding AMOR to handle structured data like SQL databases and a wider array of external tools remains a promising avenue for future research.
Summary updated