Mining How-To Text for Subgoal Labels on an Agent's Own Unlabeled Trajectories

Agents that act over many steps, such as language models operating a text simulator of a science lab or a household, learn faster when their decisions are split into two levels. A high-level "planner" picks the next subgoal ("heat the water"). A low-level "worker" issues the primitive commands that achieve it. This is hierarchical reinforcement learning. Its weak point is where the subgoals come from. Current systems either copy them from curated expert demonstrations, discover them slowly from a single end-of-task reward, or ask a language model to invent them with no check that they match what the simulator allows. Distant supervision offers a fourth source. It is the old trick of creating training labels automatically by matching unlabeled data against an existing knowledge base, as when sentences mentioning two entities are labeled with the relation a knowledge graph records between them. That trick has already turned noisy narration of YouTube videos plus wikiHow step lists into step labels for video models. Nobody has used it to label what an agent itself did.

We propose to treat an agent's own exploratory trajectories, including failed ones, the way that video work treated narrated videos. We align each trajectory, in order, to the step list of a matching how-to article. Each aligned segment becomes a subgoal-labeled training example. The resulting labels train the worker to reach a stated step and the planner to choose the next step, with no expert demonstrations and no hand-written subgoals. Distant supervision is known to produce wrong labels in both directions. We therefore clean the alignments using the worker's own estimate of whether a labeled step is reachable. The central claim is narrow and testable: the useful ingredient is grounding procedural text in the states the agent actually visited, not the procedural text alone. A language model's unaligned plan should not match it, and neither should a hierarchy that discovers subgoals from reward.

What Exists and What Is Missing

The closest supervision-side work is Learning To Recognize Procedural Activities with Distant Supervision. It matches speech-to-text transcripts of instructional videos to wikiHow steps with a sentence encoder (MPNet) and uses the matches as pseudo-labels for a video model. Those labels beat fully supervised Kinetics-400 features on COIN step classification, 54.1% versus 48.3%. We reuse its matching recipe, but that work is purely passive recognition and never trains anything to act.

Earlier combinations of reinforcement learning and distant supervision run in the opposite direction: reinforcement learning cleans distant labels, rather than distant labels training a reinforcement learner. Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning trains a flat keep-or-remove policy to filter false positives. Large Scaled Relation Extraction With Reinforcement Learning trains the extractor with bag-level rewards. H-FND adds a keep, discard, or revise decision to recover false negatives, meaning true relations the knowledge base missed. All three operate on static sentence classification. We borrow H-FND's three-way cleaning decision. What we change is the signal that drives it: the feasibility of a step for an acting agent rather than validation F1 of a classifier.

On the hierarchical side, three systems differ mainly in where their subgoals come from:

  • GLIDER seeds its planner and worker with expert trajectories, then refines them with offline and online reinforcement learning.
  • HiPER lets the agent write its own subgoals and learns them only from task reward.
  • ARM-FM prompts foundation models to write reward state machines. Its authors report that the generated completion checks often fail in complex environments because the model lacks grounding in the state space.

Our proposal removes GLIDER's demonstrations, gives HiPER an external prior over what subgoals should be, and replaces ARM-FM's ungrounded generation with alignment to visited states.

Two papers constrain the design. DIPPER shows that the worker's value function, its estimate of how likely it is to succeed from a state toward a subgoal, can screen out subgoals it cannot reach. HECRL shows that unreachable subgoals are a standard failure of generated subgoals. Both tell us that alignment noise will be fatal unless it is filtered.

Core Claim

Take the same how-to text, from wikiHow or from a language model. Converting it into segment-level subgoal labels on the agent's own trajectories, and filtering those labels by worker-estimated reachability, yields a hierarchical agent that is more sample-efficient and generalizes better to unseen task variants than three alternatives: an agent conditioned on the same text without alignment, a hierarchy that discovers subgoals from reward alone, and a flat agent. On ScienceWorld we expect it to recover most of the gap to demonstration-trained GLIDER while using no demonstrations.

Research Program

Collecting unlabeled trajectories. We collect trajectories in ScienceWorld and ALFWorld using a prompted base model plus random exploration. We withhold the benchmarks' expert demonstrations throughout. For each task family we retrieve candidate how-to articles by embedding similarity between the task description and article titles. Examples include boiling water and testing whether an object conducts electricity for ScienceWorld, and cleaning or heating an object for ALFWorld.

Aligning trajectories to steps. Each trajectory segment is represented by the text of its actions and the resulting observations. We align a trajectory $\tau$ with $T$ steps to an article's ordered step list $k_1,\dots,k_M$ by choosing boundaries that maximize total similarity under an order constraint:

$$ \max_{0=t_0<t_1<\dots<t_m\le T,\; j_1<\dots<j_m}\ \sum_{i=1}^{m}\Big[\cos\big(e(\tau_{t_{i-1}:t_i}),\,e(k_{j_i})\big)-\lambda\Big] $$

Here $e$ is the sentence encoder, $\tau_{t_{i-1}:t_i}$ is the $i$-th segment, $j_i$ indexes the step it is matched to, $m\le M$ is the number of steps matched, and $\lambda$ is a margin. The margin lets steps go unmatched, which accommodates incomplete trajectories and article steps the simulator does not support. The order constraint is the procedural analogue of the "at least once" assumption in relation extraction. It suppresses many spurious matches that a sentence-by-sentence matcher would accept.

Cleaning the labels. We first train a goal-conditioned worker on all aligned pairs. Its training signal is hindsight relabeling: segment $i$ is a successful attempt at step $k_{j_i}$. The worker's value estimate $V(s, k)$ then drives an H-FND-style decision for each label:

  • Keep a pair when the worker's value from the segment's start state is high and the segment's end state is consistent with completion.
  • Discard a pair when the value is low even after training. Such pairs are likely false positives, where the text matched but the state never changed meaningfully.
  • Revise an unmatched segment when the worker assigns high value to it for some step. These are likely false negatives that the encoder missed.

We iterate cleaning and worker training a small fixed number of times, so the labels and the worker do not chase each other indefinitely.

Training the planner and running online. The planner is trained to predict the next kept step from the current state. We then run online hierarchical reinforcement learning from task reward. We use GLIDER's offline-to-online recipe so that implementation differences do not confound the comparison. Following DIPPER, the planner's choices are regularized toward steps the worker judges reachable. The backbone is Llama-3-8B with low-rank adapters, as in GLIDER.

Measuring label quality. We measure the precision and recall of the pseudo-labels against a hand-checked sample of a few hundred segments per environment, before and after cleaning. This tells us whether downstream gains track label quality.

Baselines

All learned baselines share the backbone, the environment-interaction budget, and an equal hyperparameter-search budget, and all are evaluated on the standard seen and unseen splits.

Flat reinforcement learning. The naive reference is PPO or GRPO on task reward alone. Prior work reports that this is markedly less sample-efficient than hierarchical methods: HiPER reaches high success roughly 2.8x faster than these baselines on ALFWorld and WebShop.

Dense shaping from the same text. A second flat baseline uses the same articles without hierarchy or alignment. Its reward at each step is the maximum encoder similarity between the current observation and any article step. This isolates whether hierarchy and ordered alignment matter, or whether the text is simply a dense reward.

Reward-discovered subgoals (HiPER). HiPER is the strongest demonstration-free hierarchical competitor. It reports a 97.4% ALFWorld success rate with Qwen2.5-7B and 83.3% on WebShop. We found no reported result on Lacuna for HiPER on ScienceWorld, so we will run it there ourselves. Beating it means higher unseen-task score at equal interaction budget on ScienceWorld, and fewer samples to reach a fixed success level on ALFWorld. ALFWorld success is near ceiling, so speed is the only room to win there.

Expert-demonstration hierarchy (GLIDER). GLIDER is the strongest method overall and serves as an upper reference rather than a target to beat. It reports ScienceWorld scores of 77.43 on seen and 68.34 on unseen tasks with Llama-3-8B. Because it uses demonstrations we forbid ourselves, success means closing most of the gap between the best demonstration-free baseline and 68.34 on unseen tasks.

Ablations isolating the key ingredient.

  • Unaligned text plan: the planner is conditioned on the article's step list, or on an LLM-generated plan, and the worker is rewarded by an LLM judge of step completion, in the style of ARM-FM. This tests whether the text alone suffices without alignment.
  • LLM plan plus alignment: swaps wikiHow for LLM-generated steps inside our full pipeline. This tests whether the knowledge base matters or only the alignment does.
  • No cleaning: removes the value-based keep, discard, or revise step.
  • Shuffled steps: replaces matched steps with random steps from the same article. This tests whether the semantic content of the labels, rather than mere segmentation, drives the gains.

Why not just…?

Why not just prompt a language model for subgoals, as ARM-FM or ReAct-style agents do? This is the most serious threat. Modern language models already know how to boil water, so a wikiHow lookup may add nothing. The evidence that prompting alone falls short is indirect. ScienceWorld reports that large models with declarative knowledge average only 0.17 out of 1.0 when acting. ArCHer reports a small trained GPT-2 beating ReAct-prompted GPT-3.5 on WebShop. ARM-FM itself reports grounding failures in its generated completion checks. None of these isolates our question, which is why the "LLM plan plus alignment" and "unaligned text plan" arms exist. If an LLM plan with alignment matches wikiHow with alignment, we will report that the knowledge base is replaceable and that alignment is the contribution. If the unaligned plan matches the full method, the core claim is false.

Why not just use expert demonstrations, as GLIDER does? On these benchmarks demonstrations happen to be available, so the argument here is about settings where they are not. Labeling procedural boundaries by hand does not scale, a point the video paper makes for instructional video. The experiment reports how close a demonstration-free agent gets to GLIDER's 68.34 on unseen tasks. We will also run a small-demonstration hybrid to see whether distant labels can replace most of the expert data. That would be a smaller but still useful result.

Why not just let the agent discover its own subgoals from reward, as HiPER does? On ALFWorld this may genuinely be enough, given HiPER's 97.4%. We expect the distant labels to help mainly where exploration is hardest: ScienceWorld's longer, more varied tasks and its unseen variants. That is where reward-only discovery has the least signal to work from. If HiPER matches us on ScienceWorld at equal budget, the external prior is not earning its complexity.

Why not just turn text similarity into a dense reward for a flat agent? ARM-FM documents that dense heuristic rewards invite loopholes in which agents farm proximity reward without finishing the task. Similarity-to-text rewards seem equally exploitable, because an agent can repeatedly "look at" the kettle. The dense-shaping baseline tests this directly. We will also log reward-hacking behavior, measured as reward gained without any increase in task score.

Decisive Checks

The project succeeds if all of the following hold on ScienceWorld unseen tasks at equal interaction budget. First, the full method beats both HiPER and the unaligned text-plan arm. Second, removing cleaning measurably lowers pseudo-label precision and downstream score. Third, the shuffled-step ablation collapses toward the reward-only hierarchy. On ALFWorld, the expected win is fewer environment samples to reach HiPER's success level.

Two outcomes would falsify the thesis. One is the unaligned text plan matching the full method, which would mean alignment adds nothing. The other is the uncleaned labels matching the cleaned ones, which would mean feasibility filtering adds nothing. A further diagnostic tests the mechanism rather than the outcome: the method's advantage over the unaligned arm should grow with the fraction of article steps the simulator cannot perform, since those are exactly the steps alignment and cleaning should discard.