Learning Models of Sequential Decision-Making with Partial Specification of Agent Behavior

Vaibhav V. Unhelkar, Julie A. ShahAAAI 2019 · DOI

Abstract

Modeling how agents make sequential decisions is critical for effective human-robot interaction. However, traditional methods often fail to account for latent factors—such as changing goals or internal states—and suffer from an "alignment gap" where predicted actions do not reflect the agent's true underlying logic. This paper introduces the Agent Markov Model (AMM) and a novel Constrained Variational Inference (CVI) algorithm. By treating expert "hints" as mathematical constraints rather than raw data, the framework ensures learned models are both computationally efficient and accurately aligned with an agent's internal state transitions and policies.

Introduction

When a robot works alongside a human, it must maintain a model of the human's decision-making process. While Inverse Reinforcement Learning (IRL) is typically used to reverse-engineer goals from actions, it faces two major hurdles. First, the Alignment Gap occurs when multiple internal models explain the same behavior; a robot might predict a move correctly for the wrong reasons, leading to coordination failure in new environments. Second, Latent Factors like fatigue or shifting objectives are often unobservable, making the modeling task inherently under-determined.

Domain experts often possess partial "hints" about these latent factors (e.g., "the driver likely changed their goal at this intersection"). Most current algorithms struggle to incorporate these noisy, incomplete auxiliary inputs without breaking the underlying Bayesian mathematics. This work proposes a method to integrate such partial specifications directly into the learning process.

The Agent Markov Model (AMM)

The authors propose the Agent Markov Model (AMM) to represent the agent's state as a factored pair: a known state $x$ (e.g., position) and a latent state $s$ (e.g., a hidden goal).

The Agent Markov Model (AMM) graphical structure, illustrating the dependencies between known states (x), latent states (s), and actions (a), influenced by transition dynamics and policies.

To learn this model, the framework utilizes Bayesian Nonparametrics (specifically Hierarchical Dirichlet Processes). This allows the model to automatically discover the appropriate number of hidden states needed to explain the observed data, rather than requiring manual pre-specification.

Constrained Variational Inference (CVI)

The core technical contribution is Constrained Variational Inference (CVI). Standard inference methods treat expert knowledge as additional data points, which can lead to computational explosions. Instead, CVI treats expert "hints" as constraints on the model's parameters:

  • Local Inputs: Expert observations about specific "change points" (e.g., a goal shift at time $t$) refine the hidden state sequence.
  • Global Inputs: General domain knowledge (e.g., "this goal is rarely abandoned once chosen") constrains the transition probabilities.

By treating these as constraints, the authors preserve the efficiency of conjugate Bayesian mathematics while nudging the model toward the ground truth.

Experimental Results

The approach was evaluated in two simulated environments where ground truth was known: Line World (a 1D navigation task with goal switching) and the Highway Domain (a police car switching between nominal driving and pursuit modes).

Performance comparison across Line World and Highway domains. CVI-LG (Local and Global hints) consistently achieves the lowest error in state estimation (Hamming dist.) and policy alignment (wKL).

As shown in the results table, the full approach (CVI-LG) significantly outperformed standard Variational Inference (VI) and Maximum Entropy IRL. In the Line World, the error in estimating hidden states (Hamming distance) dropped from 0.51 to 0.13. Crucially, model alignment—measured via weighted KL divergence—was substantially improved. In the Highway Domain, CVI-LG achieved a policy alignment score of 0.33, compared to 1.04 for MaxEnt IRL. Notably, the approach remained robust even when expert hints were only 90% accurate.

Discussion and Future Work

A key insight from this work is that providing hints about latent states (where a goal changed) actually improves the model's understanding of the policy (what the agent does to reach that goal), demonstrating that information flows effectively between different model components during joint learning.

However, limitations remain. The current model identifies that distinct latent states exist but does not assign them semantic labels (e.g., "distracted" vs. "focused"). Additionally, the tabular representation used here may struggle with the continuous, high-dimensional state spaces typical of real-world robotics. Future work will focus on scaling these methods and handling multiple overlapping latent factors.

Summary updated

BibTeX
@inproceedings{unhelkar2019learning,
  author = {Vaibhav V. Unhelkar and Julie A. Shah},
  title = {{Learning Models of Sequential Decision-Making with Partial Specification of Agent Behavior}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33012522},
}
Figure