Introduction
In competitive or cooperative environments such as trading, negotiation, or strategic gaming, interactions are frequently "finitely repeated." In these scenarios, agents play a limited number of rounds against the same opponent. The primary challenge is fast adaptation: an agent must identify and counter an opponent's strategy within a very short window.
Traditional Deep Reinforcement Learning (DRL) agents often treat every new encounter as a blank slate or rely exclusively on the history of the current interaction. This approach is inefficient. If an opponent employs a strategy encountered previously, an intelligent agent should leverage that "meta-knowledge" immediately. The F3 framework is designed to bridge this gap between long-term memory and short-term adaptation.
The F3 Framework
The F3 framework consists of six integrated components that allow an agent to "recognize" strategy patterns by treating game history as a searchable library of meta-knowledge.
Opponent Action Estimator (OAE)
The OAE is the core innovation of F3. Structured similarly to a Memory Network, it observes the first few moves of a new game and retrieves $K$-similar sequences from a "Past Game History Memory." It then generates an embedding vector that predicts the opponent's future behavior, effectively performing case-based reasoning within a neural framework.
Hierarchical History Encoder (HE)
Rather than viewing actions in a flat sequence, the HE module encodes history at two distinct levels:
- Action-Level: Captures the immediate tactical responses between the agent and the opponent.
- History-Level: Captures the broader strategic trends and the overall trajectory of the interaction.
Decision Making and Fusion
The framework combines the predicted behavior from the OAE with the actual encoded history of the current game. This fused representation is passed to an Action Decoder (AD), which uses a policy gradient-based ActionNet to decide the next move. The authors found that Policy Gradient (REINFORCE) was more effective than Q-learning, as it better manages the trade-off between immediate rewards and long-term strategic positioning.
History Management
To maintain the quality of the long-term memory, a History Updater (HU) evaluates completed games and decides which are "valuable" enough to be saved, ensuring the library remains a source of useful meta-knowledge rather than noise.
Experimental Results
The F3 framework was evaluated on two classic game theory benchmarks: the Iterated Prisoner’s Dilemma (IPD) and Chicken. Using the Axelrod library, which contains over 200 distinct strategies, the researchers tested the agent's ability to generalize to "New" (unseen) strategies.
Performance and Efficiency
In the Prisoner’s Dilemma, the F3 framework achieved a score difference ($\Delta R$) of 1.116 against brand-new strategies, vastly outperforming a standard DQN (0.301) and basic Q-learning, which failed to achieve a positive score. Furthermore, F3 agents obtained significantly higher rewards within a fixed 50-turn window, demonstrating superior adaptation speed.
Cross-Game Adaptability
A key finding was the OAE's ability to generalize across different games. An OAE trained on the Prisoner’s Dilemma could be effectively reused in the game of Chicken. This suggests that the framework learns a general representation of "opponent-ness" and strategic patterns that transcend specific game rules.
Conclusion
The F3 framework demonstrates that treating past game history as a searchable library of meta-knowledge allows for rapid adaptation in repeated games. By fusing predictive embeddings with current observations, agents can defeat unknown opponents much faster than standard reinforcement learning methods. Future research will explore scaling these memory systems to handle millions of entries and extending the framework to multi-opponent environments.
Summary updated