Abstract
Real-Time Strategy (RTS) games present a significant challenge for artificial intelligence due to their high-dimensional state spaces and the need for millisecond-level decision-making. Central to an AI's performance is its evaluation function, which assesses the relative strength of game states. This paper proposes a Dynamic Adaptive Evaluation Function that combines traditional linear models with an online reinforcement learning framework and the AdamW optimizer. By treating the change in evaluation scores as a feedback signal, the system dynamically adjusts unit weights during live gameplay. Experimental results in the microRTS environment demonstrate that this approach significantly improves win rates across various map sizes and planning algorithms while maintaining a negligible computational overhead of less than 6%.
The Challenge of Real-Time Evaluation
In the fast-paced environments of RTS games, such as StarCraft or Age of Empires, an AI must constantly judge which player holds the advantage. This judgment is typically performed by an evaluation function. Historically, researchers have navigated a "trilemma" between speed, accuracy, and adaptability.
Linear methods are computationally efficient but rely on static weights that fail to capture the evolving value of units (e.g., the shifting importance of workers versus combat units). Conversely, Tree Search methods offer deep strategic exploration but are resource-intensive, and Neural Network approaches, while highly accurate, require massive datasets and significant hardware for real-time inference. The following table summarizes these trade-offs:
The core objective of this research is to bridge the gap between the speed of linear models and the adaptability of more complex systems.
Proposed Methodology: Dynamic Adaptive Evaluation
The researchers introduce a framework that makes the weights of a linear evaluation function "live." Instead of using fixed values for units, the system updates its internal valuations based on the immediate outcomes of game states. The architecture is divided into three primary modules:
- Scoring Module: This component calculates the current game score ($S_t$) and the change in score ($\Delta S_t$) from the previous time step. This delta serves as a self-supervised reward signal, indicating whether the AI's previous actions led to a better or worse position.
- AdamW Optimization: To ensure stable and efficient learning, the AdamW optimizer is integrated into the loop. It automatically manages the learning rate and weight decay, removing the need for manual hyperparameter tuning for different game scenarios.
- Online Reinforcement Learning: Using the feedback from the scoring module and the parameters from AdamW, the system performs gradient descent during the game to update unit weights ($W_{u,t+1}$). This allows the AI to realize, for instance, that defensive structures are more valuable during an enemy siege than during an expansion phase.
Experimental Setup and Results
The proposed method was rigorously tested using microRTS, a standard platform for RTS AI research. The evaluation involved a massive round-robin tournament consisting of 5,400 combat rounds across three map sizes (16×16, 24×24, and 32×32) and three distinct planning algorithms (IDABCD, IDRTMinimax, and Portfolio AI).
Key Findings
- Performance Gains: The dynamic models consistently outperformed their static counterparts. For example, using the IDRTMinimax algorithm, the Dynamic Lanchester (DL) model achieved an average score of 0.94, a substantial increase over the 0.67 achieved by the static version.
- Computational Efficiency: A critical requirement for RTS AI is low latency. The dynamic adaptation added less than 6% to the total computational overhead. In absolute terms, the additional processing time was at most 0.016ms, making it highly suitable for real-time applications.
- Complexity Scaling: Interestingly, the advantage of the dynamic method became more pronounced as map size and game complexity increased. This suggests that the self-supervised adaptation is particularly effective at managing the nuances of large-scale battlefields.
Conclusion and Future Work
This research demonstrates that online reinforcement learning can be effectively applied to linear evaluation functions to create highly adaptive RTS AI without the heavy computational costs of deep learning. By using the game's own scoring logic as a training signal, the AI achieves a form of "live learning" that responds to the specific context of every match.
Future work will focus on expanding the input features to include spatial control and economic growth rates, further refining the AI's strategic depth and its ability to handle the complex synergies of modern RTS titles.
Summary updated