Temporal Behavior Trees for Reinforcement Learning: Specifications as Rewards, Curricula, and Diagnostics

Vladimir Krsmanovic, Bernd FinkbeinerarXiv 2026 · arXiv

TL;DR

TBTCL (Temporal Behavior Tree Centric Learning) introduces a unified framework that uses a single formal object—the Temporal Behavior Tree (TBT)—to handle task specification, reward shaping, curriculum design, and policy diagnostics. By preserving the hierarchical structure of complex tasks, TBTCL significantly outperforms traditional fragmented pipelines, especially in challenging "reach-avoid" scenarios where agents must balance multiple competing objectives.

What Problem Does This Paper Address?

In specification-guided reinforcement learning (RL), researchers typically use formal logics like Linear Temporal Logic (LTL) to define what an agent should do. However, the standard pipeline is highly fragmented: the task is specified in one language, rewards are derived through another conversion (often into automata), a curriculum is manually designed, and diagnostics are performed using separate tools.

This fragmentation is problematic because crucial information—like subtask boundaries, hierarchical dependencies, and task priorities—is often lost during these intermediate conversion steps. Furthermore, standard logic-based rewards can be "sparse" or suffer from "masked gradients" (where improving one part of a task doesn't immediately improve the overall reward), making it difficult for agents to learn complex, multi-stage behaviors.

Methods

The authors propose TBTCL, which centers the entire RL process around Temporal Behavior Trees (TBTs). TBTs combine the quantitative power of Signal Temporal Logic (STL) at the "leaves" with hierarchical tree operators.

Figure 1: Temporal Behavior Trees provide a single formal object for task specification, dense robustness-based reward shaping, curriculum extraction, and trace-based diagnostics.

Key components include:

  • Unified Specification: TBTs use operators like SEQ (sequence), PAR (parallel), FBACK (fallback), and a newly introduced KLEENE (repetition) to describe complex behaviors succinctly.
  • Robustness-Based Rewards: Instead of simple binary success/failure, TBTCL derives a "dense" reward signal from the TBT's quantitative robustness. They add specific terms to ensure the agent is penalized for safety violations even after they occur and rewarded for progress in parallel tasks.
  • Automatic Curriculum Extraction: Because a TBT is a tree, every subtree is itself a valid task. TBTCL automatically extracts a Directed Acyclic Graph (DAG) of subtasks, training the agent on simpler components before tackling the full specification.
  • Trace-Based Diagnostics: The framework uses "trace segmentation" to map an agent's performance back to specific nodes in the TBT. This identifies "bottlenecks"—specific subtasks where the agent is failing—allowing the system to automatically refocus training on those weak points (Adaptive Training).

Datasets

The authors evaluated TBTCL on two primary domains across six different specifications:

  • LetterWorld: A discrete gridworld environment where an agent must navigate to lettered target cells while avoiding obstacles.
  • PointNav: A continuous 2D point-mass domain requiring precise acceleration control to reach waypoints while navigating around circular hazards.
  • Specifications: These ranged from simple reachability to complex sequential compositions (RED_THEN_BLUE), repetitions (PATROL), and multi-objective REACH_AVOID tasks.

Key Insights and Ideas

  • Succinctness over LTL: TBTs can express complex requirements much more naturally than LTL. For example, "reach S, then at least three of five targets" requires a massive, enumerated formula in LTL but is a simple tree structure in TBT.
Figure 2: Comparison of equivalent TBT and LTL specifications. Both describe the desired behavior of “reach S, and afterward at least three of the five possible targets”. While the LTL formula must explicitly enumerate structures, TBT can express them directly because of its tree structure.
  • The Structure is the Curriculum: The most elegant insight is that the hierarchical nature of the specification is the curriculum. No external logic or manual "difficulty ranking" is needed; the tree's own geometry defines the learning path.
  • Closing the Loop with Diagnostics: By using trace segmentation during training, the system doesn't just see a "low reward"; it sees exactly which leaf (e.g., "avoid_hazard") is causing the failure. This allows for "Adaptive Training," which mitigates catastrophic forgetting by revisiting specific bottlenecks.

Key Results

TBTCL was compared against several baselines, including STL Shaping, AGCL (automata-guided curriculum), and SPECTRL.

  • Superiority in Multi-Objective Tasks: On the most difficult "reach-avoid" tasks in the continuous PointNav domain, TBTCL achieved a 0.64 satisfaction rate, while all non-curriculum baselines (including standard STL shaping) remained at 0.00.
  • Efficiency: In LetterWorld's reach-avoid task, TBTCL reached a 0.85 success rate, outperforming the next best baseline (LDBA at 0.45) and standard STL shaping (0.74).
  • Complementary Benefits: Ablation studies showed that while dense rewards are enough for simple sequences, the combination of dense rewards and the automatic curriculum is essential for tasks that require balancing goal-seeking with safety constraints.

Limitations and Future Work

  • Computational Complexity: Operators like SEQ and KLEENE require searching for optimal "cut points" in a trace, which can become computationally expensive for very long trajectories.
  • Hyperparameter Sensitivity: The reward function introduces several scaling parameters (for safety, parallel progress, etc.) that require tuning to balance different task components effectively.
  • Theoretical Gaps: While the authors prove the curriculum is "well-formed," they acknowledge there are currently no formal guarantees for convergence or that the extracted curriculum order is strictly optimal.
  • Scalability: Future work is needed to test TBTCL on more complex, high-dimensional robotic tasks beyond 2D navigation.

Summary updated

Abstract

Reinforcement learning from formal specifications often separates task definition, reward shaping, curriculum design, and policy diagnostics into different mathematical objects. This fragmentation requires additional conversion steps and discards structure already present in the specification, such as subtask boundaries, ordering constraints, and quantitative measures. We introduce TBTCL (Temporal Behavior Tree Centric Learning), a framework that uses Temporal Behavior Trees (TBTs) as a unified interface for specification-guided reinforcement learning. A single TBT specifies the task, induces dense robustness-based rewards, defines a curriculum, and supports better understanding of learned policies. By adapting TBT syntax and semantics for specification-guided reinforcement learning, we provide a formalism that is more expressive than popular specification languages such as LTL and inherently hierarchical. We present algorithms to automatically extract a curriculum DAG and derive dense, robustness-based rewards directly from TBT semantics. Furthermore, we utilize TBT trace segmentation to build understandable competence profiles and automatically adapt the reward toward bottlenecks. We evaluate TBTCL on six specifications across discrete and continuous domains and show that Temporal Behavior Trees can serve as a specification, a reward, a curriculum, and a diagnostic.
BibTeX
@misc{krsmanovic2026temporal,
  author = {Vladimir Krsmanovic and Bernd Finkbeiner},
  title = {{Temporal Behavior Trees for Reinforcement Learning: Specifications as Rewards, Curricula, and Diagnostics}},
  year = {2026},
  eprint = {2611.13993},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13993},
}
Figure