Landslide and flood disasters rarely happen alone. A rainstorm triggers a debris flow, the debris dams a river, the dammed river swings against the foot of the opposite slope and erodes it, and that slope then starts to slide. When engineers ask a language model about such an event, the most useful new information usually arrives as a short field report about one link in this chain, for example "the channel has been dredged 20 m away from the landslide toe." A good answer must carry that update several steps downstream, to the slope's stability, its deformation rate, and the size of a possible outburst flood. Current hazard question-answering systems score well on direct factual questions. Nobody has measured whether a model pushes a mid-chain update through to downstream consequences, or where inside the model that push fails.
We propose to measure and repair this. The plan has three parts. We build a benchmark of paired counterfactual field updates on documented hazard chains. We use activation patching and linear probes to find the layer and token position where an update stops influencing the model's reasoning about downstream nodes. We then test whether a small intervention at that location, or a cheap monitor read from the same hidden states, beats black-box fixes such as chain-of-thought prompting and after-the-fact consistency checking.
Core Claim
Our hypothesis is that language models fail to propagate hazard-chain updates in a specific way. The update is still decodable from the hidden states at the question position, but it does not change the answer. Wang et al.'s analysis of retrieval-augmented generation predicts this: the model makes an early relevance judgment, and if the retrieved passage is judged irrelevant, the flow of context information stops (Wang et al., 2025). A downstream question such as "is the opposite slope accelerating?" shares few words with an upstream update such as "the channel was rechanneled." The model may therefore file the update as irrelevant.
If this is right, three consequences follow. Propagation failure should grow with hop distance and shrink with word overlap between update and question. Patching the update-carrying activations into the question position at the right layers should restore correct answers. A linear probe on those activations should predict failure before any text is generated. If the update is not decodable at the question position at all, the problem is information loss, not arbitration. That outcome would point to retrieval or prompting fixes rather than internal steering, and we will report it as such.
What Exists and Where This Sits
Two systems connect language models to natural hazards, and both treat the model as a black box. MoRA-RAG splits geotechnical reconnaissance reports into small factual statements, routes each query to hazard-specific document collections, and runs a five-agent verification loop. It reports 94.5% accuracy on the 5,776-question HazardRecQA benchmark at about 20 seconds per query (Kuai et al., 2025). WildfireGPT chains planning and analysis agents over climate data. Its authors report that agents get distracted and misread inputs, but they have no internal diagnostic to explain why (WildfireGPT). Both motivate this proposal: they show that task accuracy on direct questions is high while leaving multi-hop update behavior unmeasured. We do not build a better retrieval system. We study what the model does with a retrieved update once it is in the context window.
The physical side supplies the ground truth for which way each downstream quantity should move. Lan et al. (2022) document the Meilong chain end to end. Moving the river 20 m restored lateral support at the toe, and peak deformation dropped from 75 mm/h to 8.74 mm/h. Without the intervention, a simplified slice-based stability calculation (the Janbu method) gives a factor of safety of 0.93, where the factor of safety is the ratio of resisting to driving forces and values of 1.0 or below mean failure. Lan et al. also document a hydraulic lag between peak river discharge and slope response. Chen et al. (2022) add freeze-thaw timing of river-damming landslides. Gu et al. rank debris-flow drivers in the Jinsha basin as erodibility, then sediment connectivity, then stream power, then extreme rainfall. The earthquake cascade model online-DisasterVINF encodes explicit causal graphs linking shaking to landslides, liquefaction, and building damage, and it updates them as field data arrive (online-DisasterVINF). These papers constrain our labels: we only score directions of change that the physical literature fixes.
The interpretability side supplies the methods. Zhu et al. show that linearly probed directions for numbers are causally used by the model, with interventions succeeding up to 0.73 of the time. They also find that exact numerical recovery stays below 50%. That warning is why we score directions of change rather than numeric safety-factor values. Wang et al., 2025 supply a four-stage account of how context is used across layers and a neuron-selection method, Knowledge Activation Probability Entropy (KAPE). Switching off the "external knowledge" neurons it selects costs about 11.36 F1 when the passage is correct and gains 0.78 F1 when the passage is noisy. We borrow this as a second, neuron-level intervention. Zhang et al. find that a few early layers carry outsized importance. This tells us to scan all layers rather than assume mid-depth. Two papers find that model-written explanations are often unfaithful to the computation behind them (Mahon et al., 2026; Wadhwa et al., 2026). This bears directly on whether chain-of-thought counts as a fix.
Research Program
The benchmark. We will build HazardChain-Update from documented chains. The sources are the Meilong, Qilian, and Jinsha case studies and the cascade structures in online-DisasterVINF's Haiti, Puerto Rico, and Hokkaido events. We will also use GEER-derived event descriptions from HazardRecQA, where the question sets permit, to add chains from the 90 covered events.
Each item has four parts. The first is a base reconnaissance context. The second is an update about node $i$ of a chain. The third is a question about node $i+k$, for $k$ from 0 to 3. The fourth is an answer label of increase, decrease, or unchanged. Every update comes in a matched pair that flips the downstream direction, such as "channel pushed against the toe" versus "channel moved away from the toe." We will choose pairs where at least one member contradicts the model's no-update default, so that prior knowledge alone cannot score above chance.
We will also add distractor updates that a competent engineer should ignore, such as a report from a different valley. These test whether a repair simply makes the model gullible. A geotechnical annotator will verify each label against the source paper. We target about 1,500 pairs, and items where the physics is ambiguous are dropped rather than guessed.
Models. We use LLaMA-3-8B and Mistral-7B, because the cited interpretability results were obtained on these families, and LLaMA-3-70B to check scale.
Localization. For each pair, we run the two members and patch residual-stream activations from one into the other. We patch at the update tokens and at the question's final token, layer by layer. We record which layer-position combinations flip the answer. Separately, we train a linear probe to decode the update's identity from the question-position hidden state at each layer. Crossing probe decodability with answer correctness separates two cases: "present but unused," which is our hypothesis, and "lost."
Repair and monitoring. Two repairs are tested. The first adds the probed update direction at the question position in the layers identified by patching. The second amplifies KAPE-selected external-knowledge neurons. A third product is a monitor: a probe on the same hidden states that predicts, before any tokens are generated, whether the answer will ignore the update.
Baselines
The naive reference point is the base model answering with the update simply present in its context and no special instruction. This gives the size of the propagation gap: accuracy at $k=0$ minus accuracy at $k=2$ or $3$. Any method must close this gap without lowering $k=0$ accuracy or accuracy on distractor items.
The strongest existing system is MoRA-RAG's agentic loop, which reports 94.5% accuracy on HazardRecQA at about 20 seconds per query (Kuai et al., 2025). That number comes from direct questions, not update propagation. We found no result on Lacuna for multi-hop update propagation on any hazard benchmark. To compare fairly, we will re-implement MoRA-RAG's evaluator-and-rewriter loop on the same base models, give it the same context, and report accuracy and wall-clock latency. Beating it means matching its downstream-hop accuracy at a small fraction of its latency. Alternatively, it means exceeding its accuracy at $k \ge 2$ if the agents also fail to propagate updates.
Chain-of-thought prompting, which asks the model to walk the chain step by step, is the strongest cheap competitor. It runs on the same models with the same prompts and an equal budget for prompt tuning on a held-out development split.
For the monitor, the baselines are the model's own answer-token probability and an after-the-fact textual-entailment checker between update and answer. The monitor must beat both on AUROC for predicting wrong answers.
An oracle upper bound comes from hand-encoding each benchmark chain as a small causal graph in the style of online-DisasterVINF and propagating the extracted update symbolically.
Two ablations isolate the key ingredient, which is intervening at the patching-identified location. The first applies the same steering at randomly chosen layers. The second uses a probe trained only on input token embeddings, which tests whether the monitor merely detects surface cues like the word "dredged."
Why not just…?
Why not just evaluate behavior on the counterfactual pairs and stop there? Behavioral pairs are the measurement here, and they alone will establish whether a propagation gap exists. They cannot say whether the update was dropped or held but unused, and those two failures need different fixes: better retrieval and prompting for the first, internal steering for the second. They also give no signal at deployment time, when there is no answer key. The probe-versus-correctness analysis is the experiment that shows what behavior misses.
Why not just use chain-of-thought? This is a real threat. If chain-of-thought closes the gap at $k=3$, the internal repair is only worth its latency savings and we will say so. Two things keep the comparison open. Explanations are often unfaithful to the computation behind them (Mahon et al., 2026), so a correct-looking walk through the chain can end in an answer the walk does not support. We measure this directly by checking whether the final answer agrees with the model's own stated intermediate steps. Chain-of-thought may also make the model accept distractor updates more readily, which the distractor split will reveal.
Why not just let a causal graph do the propagation and use the LLM only to extract facts? For chains that someone has already encoded, this is probably better, and we include it as an oracle for exactly that reason. Its limit is coverage. online-DisasterVINF needs its causal structure and physical pruning rules specified in advance (online-DisasterVINF). Reconnaissance reports routinely describe chains that nobody has encoded, such as the freeze-thaw damming in Chen et al. (2022). We will report results separately on chains held out from any hand-built graph. That is where the LLM's own propagation matters.
Why not just check answers against the update with a textual-entailment model? Entailment checks agreement with what the update literally says. Downstream consequences are, by construction, not stated in the update, so a correct $k=2$ answer and a wrong one can look equally compatible with it. The monitor comparison at $k \ge 2$ will show whether this intuition holds.
Decisive Checks
The study succeeds if four things hold. First, the propagation gap exists and grows with hop count. Second, failure rate falls as word overlap between update and question rises, which is the relevance-gating signature predicted from Wang et al., 2025. Third, a large share of failures fall in the present-but-unused cell. Fourth, steering at the patched layers recovers downstream accuracy while random-layer steering does not, without costing accuracy on direct or distractor items.
The main claim fails if the update is usually absent from the question-position hidden states, or if chain-of-thought matches the repair on both accuracy and faithfulness. In either case, the contribution reduces to the benchmark and a negative result on internal repair. That result is still useful to anyone deploying these systems in an emergency response.
References
- Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation
- Knowledge-Grounded Agentic Large Language Models for Multi-Hazard Understanding from Reconnaissance Reports
- A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptation
- Multi-Hazard Chain Reaction Initiated by the 2020 Meilong Debris Flow in the Dadu River, Southwest China
- Activity and Kinematics of Two Adjacent Freeze-Thaw-Related Landslides Revealed by Multisource Remote Sensing of Qilian Mountain
- Debris flow susceptibility in the Jinsha River Basin, China: a Bayesian assessment framework based on geomorphodynamic parameters
- Scalable Variational Learning for Noisy-OR Bayesian Networks with Normalizing Flows for Complex Cascading Disaster Systems
- Language Models Encode the Value of Numbers Linearly
- Investigating Layer Importance in Large Language Models
- A Definition of Good Explanations and the Challenges Explaining LLM Outputs
- ART: Adaptive Reasoning Trees for Explainable Claim Verification