Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains. However, they frequently encounter limitations when faced with complex, multi-disciplinary reasoning tasks—such as medical entrance exams that require simultaneous mastery of biology, chemistry, and ethics. Current approaches to this problem generally fall into two categories: coarse-grained routing, which sends a query to a single "best" expert model, and Mixture-of-Agents (MoA), which involves unstructured debate or aggregation among multiple models.
As illustrated in Figure 1, traditional methods often fail to capture the nuanced interdependencies between subjects within a single question. The authors of S-DAG argue that effective reasoning requires fine-grained, subject-level coordination, focusing not just on agent selection but on the logical flow of information between specialized experts.
The S-DAG Framework
The S-DAG (Subject-Based Directed Acyclic Graph) framework introduces a structured approach to multi-agent collaboration through a two-stage process: question analysis and graph-based execution.
1. Question Analysis and DAG Generation
Rather than relying on inconsistent LLM self-prompting to determine task structure, S-DAG utilizes a trained Graph Neural Network (GNN). When a query is received, the GNN identifies the relevant subjects and predicts their dependencies. It constructs a Directed Acyclic Graph where nodes represent specific subjects and edges define the information flow. For instance, a "supporting" subject like Mathematics may be solved first to provide necessary context for a "dominant" subject like Physics.
2. Multi-Agent Collaboration and Execution
The system manages a pool of 14 specialized, lightweight LLMs (typically 7B to 13B parameters). By maintaining a "Capability Profile" for each model, S-DAG matches graph nodes to the most proficient experts. Supporting agents generate intermediate reasoning, which is passed as context to subsequent agents in the graph, eventually reaching the "Dominant Agent" for the final answer.
Experimental Setup and Datasets
To evaluate S-DAG, the researchers curated multi-subject subsets from three rigorous benchmarks where questions averaged between 3.5 and 4.4 distinct subjects:
- MMLU-Pro: College-level multi-disciplinary subjects.
- GPQA: Graduate-level science questions.
- MedMCQA: Medical entrance examination queries.
Key Results and Insights
Performance and Efficiency
S-DAG achieved an average accuracy of 59.73% across the benchmarks, outperforming both the Symbolic-MoE baseline (57.53%) and the closed-source GPT-4o-mini (58.52%). Remarkably, this performance was achieved using a collection of small models, rivaling much larger generalist models like Qwen2.5-72B and Llama3.3-70B.
Beyond accuracy, S-DAG proved significantly more efficient than fully connected multi-agent setups. By restricting communication to the logical paths defined by the DAG, the system reduced LLM calls by 50% and cut inference time from 38.45 seconds to 15.02 seconds.
The Importance of Structure
A critical finding of the study is that "more communication" is not always better. Fully connected graphs, where every agent interacts with every other agent, often performed worse due to the introduction of reasoning loops and redundant "noise." The GNN-generated DAG acts as a traffic controller, ensuring that information flows only where it is logically required.
Conclusion and Future Work
S-DAG demonstrates that structured coordination can effectively compensate for raw parameter count in complex reasoning tasks. While the framework shows great promise, its performance remains dependent on the quality of the underlying expert pool. Future research will explore dynamic expert fetching and the implementation of safety safeguards for high-stakes applications like medical diagnosis, where error propagation must be strictly controlled.
Summary updated