The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learning

Calarina Muslimani, Yunshu Du, Kenta Kawamoto, Kaushik Subramanian, Peter Stone, Peter R. Wurman

Abstract

The success of reinforcement learning (RL) is fundamentally tied to having a reward function that accurately reflects the task objective. Yet, designing reward functions is notoriously time-consuming and prone to misspecification. To address this issue, our first goal is to understand how to support RL practitioners in specifying appropriate weights for a reward function. We leverage the Trajectory Alignment Coefficient (TAC), a metric that evaluates how closely a reward function's induced preferences match those of a domain expert. To evaluate whether TAC provides effective support in practice, we conducted a human-subject study in which RL practitioners tuned reward weights for Lunar Lander. We found that providing TAC during reward tuning led participants to produce more performant reward functions and report lower cognitive workload relative to standard tuning without TAC. However, the study also underscored that manual reward design, even with TAC, remains labor-intensive. This limitation motivated our second goal: to learn a reward model that maximizes TAC directly. Specifically, we propose Soft-TAC, a differentiable approximation of TAC that can be used as a loss function to train reward models from human preference data. Validated in the racing simulator Gran Turismo 7, reward models trained using Soft-TAC successfully captured preference-specific objectives, resulting in policies with qualitatively more distinct behaviors than models trained with standard Cross-Entropy loss. This work demonstrates that TAC can serve as both a practical tool for guiding reward tuning and a reward learning objective in complex domains.

Introduction

In reinforcement learning, the reward function serves as the agent's primary objective. However, designing these functions is a grueling process of trial and error. Slight misspecifications often lead to "reward hacking," where agents achieve high scores through unintended or dangerous behaviors. This challenge is particularly acute in preference-heavy domains like autonomous driving, where subjective styles (e.g., "aggressive" vs. "timid") must be captured. This paper addresses two critical gaps: the lack of clear metrics for manual tuning and the brittleness of standard loss functions used to learn rewards from human preferences.

Act 1: Manual Reward Tuning with TAC

The authors introduce the Trajectory Alignment Coefficient (TAC), a metric based on Kendall’s Tau-b that measures the correlation between two rankings of trajectories: one from a human expert and one induced by a candidate reward function.

In a human-subject study using the Lunar Lander environment, RL practitioners were provided with real-time TAC scores as they adjusted reward weights. This feedback acted as a "hot/cold" signal for alignment. The results were significant: practitioners with access to TAC produced agents with a 62.7% success rate, compared to just 19.8% for the control group who only saw raw reward totals.

Figure 1: Average success rate in Lunar Lander comparing TAC-guided alignment vs. control.

Beyond performance, TAC reduced the perceived cognitive workload of reward tuning by approximately 30%, transforming a blind guessing game into a guided search.

Act 2: Automated Learning via Soft-TAC

To automate the alignment process, the researchers developed Soft-TAC, a differentiable version of the metric. By employing a tanh relaxation, TAC is transformed into a loss function optimizable via gradient descent.

Unlike standard Cross-Entropy loss, Soft-TAC is inherently noise-tolerant. When faced with contradictory or noisy human preferences, the tanh function saturates, effectively ignoring outliers rather than allowing them to distort the reward model. This robustness allows the model to focus on the consistent majority of the preference data.

Experimental Results in Gran Turismo 7

The efficacy of Soft-TAC was tested in Gran Turismo 7 (GT7), a high-fidelity racing simulator. The researchers sought to learn rewards for specific driving styles, such as "Aggressive" racing.

As shown in the performance metrics, Soft-TAC agents successfully captured the intended style. For aggressive preferences, Soft-TAC agents exhibited significantly higher collision rates and off-track instances (corner cutting) to gain advantages, resulting in more distinct behavioral profiles than those trained with standard Cross-Entropy loss, which often converged toward a generic middle ground.

Figure 4: Performance metrics in GT7 for aggressive preferences, showing Soft-TAC's ability to capture distinct behaviors.

In time-trial benchmarks, Soft-TAC learned a "progress" weight that led to lap times 7 seconds faster than Cross-Entropy and 5.5 seconds faster than the game's default reward.

Discussion and Limitations

While TAC and Soft-TAC represent significant steps forward, limitations remain. The current implementation focuses on linear reward functions (weighting pre-defined features) rather than complex neural network reward models. Additionally, the system is dependent on the quality of the initial feature set; TAC cannot "invent" a missing critical feature. Future work may explore using TAC to identify missing features, potentially creating an automated feature discovery loop.

In conclusion, TAC serves as both a practical tool for human-in-the-loop tuning and a robust objective for automated reward learning in complex, preference-driven environments.

Summary updated

Abstract

The success of reinforcement learning (RL) is fundamentally tied to having a reward function that accurately reflects the task objective. Yet, designing reward functions is notoriously time-consuming and prone to misspecification. To address this issue, our first goal is to understand how to support RL practitioners in specifying appropriate weights for a reward function. We leverage the Trajectory Alignment Coefficient (TAC), a metric that evaluates how closely a reward function's induced preferences match those of a domain expert. To evaluate whether TAC provides effective support in practice, we conducted a human-subject study in which RL practitioners tuned reward weights for Lunar Lander. We found that providing TAC during reward tuning led participants to produce more performant reward functions and report lower cognitive workload relative to standard tuning without TAC. However, the study also underscored that manual reward design, even with TAC, remains labor-intensive. This limitation motivated our second goal: to learn a reward model that maximizes TAC directly. Specifically, we propose Soft-TAC, a differentiable approximation of TAC that can be used as a loss function to train reward models from human preference data. Validated in the racing simulator Gran Turismo 7, reward models trained using Soft-TAC successfully captured preference-specific objectives, resulting in policies with qualitatively more distinct behaviors than models trained with standard Cross-Entropy loss. This work demonstrates that TAC can serve as both a practical tool for guiding reward tuning and a reward learning objective in complex domains.
BibTeX
@misc{muslimani2026trajectory,
  author = {Calarina Muslimani and Yunshu Du and Kenta Kawamoto and Kaushik Subramanian and Peter Stone and Peter R. Wurman},
  title = {{The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learning}},
  year = {2026},
  eprint = {2601.16906},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.16906},
}
Figure