Many robot tasks only make sense with memory. A robot can place a cake in the oven only after the batter has been mixed. A gripper may tilt a cup only once the cup is empty. A common way to encode such tasks is a reward machine, a small finite-state automaton that reads high-level events ("cup grasped", "door open") and keeps track of which stage of the task the robot is in. Every method in this line of work assumes those events come from a perfect detector. On real hardware they come from noisy cameras and LiDAR. One missed or spurious event pushes the automaton into the wrong stage. From then on the robot follows the wrong sub-policy and obeys the wrong safety rules, and nothing in the system notices.
We propose to treat the automaton stage as a hidden quantity to be inferred. The robot keeps a probability distribution over which stage it is in, which we call a belief, and updates it from calibrated detector outputs and a model of how sensors fail, including bursts of correlated failure. It then acts on that belief. An imitation-learned policy is conditioned on the belief. A safety filter enforces the constraints of every stage that is still plausible. When the belief is too spread out, the robot pauses or backs off instead of acting on a guess. The claim is that under realistic sensor degradation, losing track of the task stage is a major and largely unmeasured cause of both task failure and safety violations in multi-stage imitation. We also claim this failure is fixable without retraining the perception stack.
Core Claim
Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning introduced reward machines and showed that pairing the environment state with the automaton state makes a history-dependent task memoryless again. LTL and Beyond showed that temporal-logic specifications compile into such machines. Learning Reward Machines for Partially Observable Reinforcement Learning learns the machine from experience. All three supply the formal object we reuse. All three also assume a noiseless labeling function, the map from observations to events. We remove that assumption.
The closest connecting work is Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints (SafeSAC-H). It summarizes history with a recurrent network trained on 400,000 labeled trajectories and tunes a Lagrange multiplier. It has no explicit task structure, no sensor-noise model, and no demonstrations. Our proposal differs on three points. It keeps the explicit automaton. It replaces hard event labels with a two-level hidden-state filter over pairs of (automaton stage, sensor condition). It uses the resulting uncertainty about the stage as the trigger for a safety filter and for imitation recovery.
The filter update is
$$b_{t+1}(u', m') \propto \sum_{u, m} b_t(u, m)\, P(m' \mid m) \sum_{\sigma} P(\sigma \mid o_{t+1}, m')\, \mathbb{1}[\delta(u, \sigma) = u'].$$
Here $b_t$ is the belief at time $t$, $u$ is the automaton stage, and $m$ is a discrete sensor condition (nominal or degraded) with Markov switching $P(m' \mid m)$. $\sigma$ is a set of events, $P(\sigma \mid o, m)$ is the calibrated detector likelihood given observation $o$, and $\delta$ is the automaton's transition function.
The sensor-condition variable is what lets the filter discount a burst of blur or LiDAR dropout as a whole. Without it, a burst looks like many independent pieces of evidence that the stage has changed.
Research Program
The system has four parts that we build and analyze separately.
Event detectors. These are trained in simulation with noise injection. They are then calibrated per sensor condition on held-out corrupted data, so that $P(\sigma \mid o, m)$ means what it says.
Belief-conditioned policy. This is a behavior-cloned policy with one head per automaton stage, trained on demonstrations labeled with the true stage. At run time the heads are combined by belief weight.
Belief-level safety filter. Each automaton stage carries its own constraints, such as a force limit while holding an object or a keep-out zone before a door is opened. The filter enforces the union of constraints over all stages with belief mass above a threshold $\eta$. When the belief entropy exceeds a second threshold, control hands over to a stage-agnostic hold-or-retreat behavior also cloned from demonstrations.
Drift coupling. This part addresses an interaction we expect to matter. When an imitation policy drifts off the demonstrated distribution, a problem documented by Learning Monocular Reactive UAV Control in Cluttered Natural Environments and Addressing Limitations of State-Aware Imitation Learning for Autonomous Driving, the detectors also see unfamiliar images and become less reliable. We therefore borrow the ensemble-disagreement novelty signal from ThriftyDAgger and use it to raise the prior probability of the degraded sensor condition. Policy drift then automatically makes stage tracking more cautious.
The analysis component measures desynchronization. Desync rate is the fraction of timesteps where the most likely stage under the belief differs from the true stage. Resync time is how long the system takes to recover. We will break down task failures and safety violations by whether a desync preceded them. This breakdown tests the first half of the core claim directly.
Baselines
Naive reference. The same stage-conditioned behavior-cloning policy, run with the original reward-machine pipeline from Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning: detector outputs thresholded at 0.5 and the automaton advanced deterministically. Safety constraints are enforced only for the current stage. This baseline shows how much damage label noise does when nothing is done about it.
Debounced labels. The same pipeline, but an event is accepted only after it is detected in $k$ of the last $n$ frames, with hysteresis for clearing. This is the engineer's obvious fix. We tune $k$ and $n$ on the same validation noise as our filter's thresholds.
Recurrent history baseline. We consider this the strongest existing method. It follows the SafeSAC-H design of Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints, adapted to imitation. A GRU (a recurrent network) over observations replaces the explicit automaton, and it is trained on the same demonstrations with the same noise augmentation. A recurrent safety-cost model, trained on the same labeled rollouts our filter uses for calibration, drives a Lagrangian penalty. It gets the same demonstration count, gradient steps, and hyperparameter search budget as our method.
SafeSAC-H reports that it "consistently met the target of 90% safe trajectories across almost all tasks" on MuJoCo, Bullet Safety Gym, and RDDL Gym with clean observations. No reported result found on Lacuna for any method on reward-machine tasks with noisy event detectors. All numbers in our noisy settings will be measured fresh. Beating this baseline means a lower safety-violation rate at equal or higher task success, under bursty noise, with non-overlapping confidence intervals over seeds.
Ablations. Three ablations isolate the key ingredients. The first removes the sensor-condition variable, giving a single-level filter that treats detector errors as independent across frames. The second keeps the full filter but enforces only the most likely stage's constraints. The third keeps the filter and safety rule but removes the drift coupling.
Why not just…?
Why not just debounce the detectors? For independent per-frame flicker, debouncing may work about as well as our filter, and we expect to find that. It fails in two regimes. The first is a long correlated burst, such as motion blur during a fast motion or an occlusion of the kind discussed by Jang et al.. A burst outlasts any window short enough to keep reaction time acceptable, so either the burst gets through or genuine events are delayed. The second is that debouncing outputs a hard stage. It cannot express "I am unsure whether the cup is empty," so the safety filter cannot hedge. Our noise sweep varies burst length while holding the total error rate fixed. If debouncing matches us at all burst lengths, the sensor-condition model is unnecessary and we will say so.
Why not just use a recurrent policy and drop the automaton? SafeSAC-H shows that recurrent summaries can carry history-dependent safety. However, Learning Reward Machines for Partially Observable Reinforcement Learning reports that LSTM-based A3C, PPO and ACER failed on the Cookie, Symbol and 2-Keys memory tasks where an explicit machine succeeded. A recurrent state also offers no inspectable stage on which to attach stage-specific constraints. This is a real competitor, especially with heavy noise augmentation. The head-to-head on matched data settles it.
Why not just make perception robust, by abstracting features or compressing to a latent? Deep Drone Acrobatics and On the use of Deep Autoencoders for Efficient Embedded Reinforcement Learning show that better input representations narrow the sim-to-real gap. The drone paper also reports total failure once the upstream tracker collapses. Better detectors reduce how often errors happen, but they do not prevent occasional errors from becoming permanent. That is a structural problem with discrete transitions. We run all methods with the same noise-augmented detector so the comparison measures what happens downstream of perception, and we also report results with clean detectors.
Why not test-time search like SAILOR? A Smooth Sea Never Made a Skilled SAILOR recovers from drift by searching in a learned world model. It assumes a memoryless reward, enforces no constraints, and is expensive at inference. It is complementary rather than competing, so we leave it out of the main comparison.
Decisive Experiments
Office World and Minecraft World. We first use the discrete reward-machine domains from Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning and LTL and Beyond. We add synthetic label noise of three kinds: independent flips, bursty dropouts, and false positives correlated with a nuisance state. Hazard cells carry stage-dependent costs. This is where the desync breakdown and the burst-length sweep are cleanest.
Safety-Gymnasium Button. Next we use the Button tasks from Safety Gymnasium, adding an ordered-button specification compiled to an automaton, vision observations corrupted with blur and dropout, and a scripted expert for demonstrations. As a reference point for the Lagrangian approach, Safety Gymnasium reports that PPO-Lag on velocity tasks achieved a "98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward". This is on clean, memoryless tasks.
Robosuite. Finally we use the multi-stage Stack and BreadCanRandom tasks from the suite used by SAILOR, with camera corruption and per-stage force and workspace constraints.
Metrics. Across all three settings we measure task success, safety-violation rate, desync rate, and resync time.
What would support or kill the claim. Three outcomes would support it: (1) under bursty noise, most violations in the naive and debounced baselines are preceded by a desync; (2) the full method cuts violations relative to the recurrent baseline at matched success; (3) removing the sensor-condition variable hurts mainly at long burst lengths. Three outcomes would kill it or narrow it substantially: desync explains few failures; debouncing matches the full method at every burst length; or enforcing constraints for all plausible stages makes the robot so cautious that task success collapses.
References
- Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning
- LTL and Beyond: Formal Languages for Reward Function Specification in Reinforcement Learning
- Learning Reward Machines for Partially Observable Reinforcement Learning
- Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
- Learning monocular reactive UAV control in cluttered natural environments
- Addressing Limitations of State-Aware Imitation Learning for Autonomous Driving
- ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
- ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context
- Deep Drone Acrobatics (Extended Abstract)
- On the use of Deep Autoencoders for Efficient Embedded Reinforcement Learning
- A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark