Abstract
An advising agent, a coach, provides advice to other agents about how to act. In this paper we contribute an advice generation method using observations of agents acting in an environment. Given an abstract state definition and partially specified abstract actions, the algorithm extracts a Markov Chain, infers a Markov Decision Process, and then solves the MDP (given an arbitrary reward signal) to generate advice. We evaluate our work in a simulated robot soccer environment and experimental results show improved agent performance when using the advice generated from the MDP for both a sub-task and the full soccer game.
Introduction
In complex multi-agent systems, the role of a "coach" is to observe agent behavior, identify successful strategies, and communicate improvements. However, automated coaching faces significant hurdles. First is Action Blindness: coaches often observe the outcomes of actions (e.g., a ball moving) without access to the internal commands or intent of the agents. Second is the State Space Explosion: continuous environments like soccer fields present infinite possible states, making standard reinforcement learning intractable. Finally, Communication Constraints necessitate that advice be high-level and strategic rather than low-level micromanagement. This paper proposes a framework to overcome these challenges by learning an abstract Markov Decision Process (MDP) from raw observations.
The Advice Generation Pipeline
The proposed method transforms raw observation data into actionable advice through a structured pipeline. It begins with State Abstraction, where raw coordinates are mapped to discrete "abstract states" using domain knowledge, such as grid locations on a soccer pitch or player occupancy regions. From these sequences, a Markov Chain is extracted to represent transition probabilities between abstract states.
As shown in Figure 1, the pipeline integrates abstract state definitions and observation data to produce traces. These traces are combined to form a Markov Chain, which is then augmented with actions and rewards to produce a final MDP. This modular approach allows the coach to generate different strategies by simply altering the reward signal without needing new observation data.
Inferring the MDP from Transitions
The core innovation lies in converting the learned Markov Chain into an MDP. Since the coach cannot see the agents' actions, it uses a library of "abstract actions" (e.g., passing, dribbling). Each action is defined by its primary (intended) and secondary (unintended or error-prone) effects.
Figure 2 illustrates this process:
- Markov Chain (a): Captures observed transition probabilities between states (e.g., $s_0 \to s_1$ with $p=0.6$).
- Action Association (b): Observed transitions are matched against a library of abstract actions ($a_0, a_1, a_2$). Actions are defined by their primary effects (thick arrows) and secondary effects (thin arrows).
- MDP Construction (c): The algorithm labels transitions with the most likely actions, creating a model where transitions are conditioned on agent choices.
Once the MDP is constructed, the coach assigns a reward signal (e.g., +100 for a goal) and solves the MDP using dynamic programming. The resulting optimal policy is translated into CLang, a standard rule-based advice language used in the RoboCup community.
Experimental Evaluation
The system was evaluated in the RoboCup Simulation League's Soccer Server System using data from international competitions and internal experiments.
- Circle Passing Sub-game: To isolate variables, the coach observed agents performing a circular passing drill. The generated advice increased the success rate of reaching target cells from 40% to 88%.
- Full Game Performance: The coach observed one team (UTA) and provided advice to another (CM Wyverns). In 11-vs-11 matches against a fixed opponent, the advised team improved its average score difference by 1.6 goals per game, a statistically significant result ($p < 0.01$).
Conclusion and Future Work
This research demonstrates that high-level strategic advice can be generated purely from observation by leveraging state and action abstraction. While the current model relies on manually defined abstract states and action effects, it successfully bridges the gap between raw data and strategic guidance. Future work includes explicitly modeling opponent behavior to refine strategies and using smoothing techniques to address data sparsity in less-visited regions of the state space.
Summary updated