Abstract
Evaluating autonomous trading agents in the Trading Agent Competition for Supply Chain Management (TAC-SCM) is challenging due to the stochastic nature of opponent behavior. This paper presents a benchmarking framework using specialized "Market Relief" and "Market Pressure" agents to simulate controlled economic environments. By varying market pressure, researchers can characterize an agent's robustness and identify critical failure points. Experiments with the MinneTAC agent reveal specific thresholds where profit margins collapse and material costs exceed revenue, providing a quantitative basis for agent optimization.
Introduction
In the Trading Agent Competition for Supply Chain Management (TAC-SCM), agents must navigate a complex supply chain environment, managing procurement, production, and sales in a fast-paced simulation. A significant hurdle in agent development is the lack of stable, universal benchmarking. Because an agent's performance is heavily influenced by the unpredictable strategies of five other competitors, high profits may result from a "weak" field rather than superior strategy. This research addresses the need for repeatable, scientific experiments to characterize agent performance across a spectrum of market conditions.
Benchmarking Methodology
The authors introduce two specialized "benchmark agents" designed to manipulate market dynamics from within the simulation:
- The Market Relief Agent: A passive agent that occupies a slot but performs no transactions. This reduces demand for parts and supply of finished goods, creating a "low-pressure" environment with lower costs and higher selling prices.
- The Market Pressure Agent: A configurable "bully" agent with unlimited credit. It is programmed to capture a specific market share percentage by aggressively outbidding competitors, thereby driving up material costs and depressing customer prices.
By substituting standard competitors with these agents, developers can create a "configurable level of pressure" to stress-test their target agent's strategy.
Experimental Evaluation
The study utilized the TAC-SCM simulation environment to evaluate MinneTAC, a prominent agent from the University of Minnesota. By varying the number and intensity of Market Pressure agents, the researchers mapped MinneTAC's performance across a wide range of economic climates.
Performance Thresholds and Insights
The experimental results revealed several critical performance characteristics of the MinneTAC agent as market pressure increased.
As shown in the figure above, the agent's performance is not linear. Several key insights emerged:
- Profit Slope as a Robustness Metric: Robustness is defined by a shallow profit slope. A "better" agent maintains profitability longer as pressure increases.
- The 70% Inflection Point: At approximately 70% market pressure, the profit slope for MinneTAC changes abruptly, indicating a breakdown in strategic efficiency.
- Market Saturation Threshold: At 83% market pressure (equivalent to 5 out of 6 slots being dominated by aggressive agents), the market reaches a saturation threshold where price wars become unavoidable.
- Negative Profitability: Beyond 92% market pressure, material costs surpass revenue. At this point, the agent begins losing money on every transaction, suggesting that a sophisticated agent should recognize these conditions and cease trading.
Conclusion and Future Work
The proposed benchmarking tools provide a quantitative method for identifying the "breaking points" of autonomous trading strategies. While effective at simulating price and quantity pressure, the current Market Pressure Agent does not yet account for the temporal dimension of competition, such as delivery dates. Future research will focus on incorporating multi-dimensional competition to more accurately simulate long-term strategic pressures in the parts market.
Summary updated