Do Language Models Assume Lift Always Falls? Testing Monotone-Physics Bias with the Cavitation Lift Bump

When a hydrofoil (an underwater wing) moves fast enough, the pressure over its surface drops below the point where water boils, and pockets of vapor form. This is cavitation. The textbook story is that more cavitation means less lift. The measured story is different. On a Clark-Y hydrofoil at a small angle of attack, lift first rises as a vapor sheet grows, peaking near cavitation number $\sigma \approx 0.8$, and only then collapses. The cavitation number is a dimensionless measure of how close the flow is to boiling; lower means more vapor. The rise, called the "lift bump," happens because the vapor sheet bends the streamlines and makes the foil behave as if it were more curved, while vortices shed from the sheet eventually destroy lift (Okabayashi et al., LES of cavitating Clark-Y flow). Standard averaged-turbulence simulations miss the bump entirely. We propose to use this effect, a response that goes up before it comes down because two mechanisms compete, as a controlled probe of a specific failure in large language models (LLMs). The hypothesis is that LLMs asked to reason about physical systems default to "more of the bad thing, more of the bad outcome," and that this monotone prior overrides mechanisms they were explicitly given.

The proposal has two parts. The first is a benchmark, CavLift-Probe, that separates recall of the lift bump, derivation of non-monotone behavior from stated mechanisms, and consistency between hard and easy versions of the same question. The benchmark includes matched items where the monotone answer is actually correct, so a model that always predicts a bump gets no credit. The second part tests one intervention, which we call mechanism-isolated decomposition. The model computes each competing mechanism's contribution in a separate context, runs code on it, and only then combines the pieces. This tests whether isolating the mechanisms, rather than extra computation, is what removes the bias.

Core Claim

We claim that LLM errors on non-monotone physical responses are asymmetric. Models predict monotone behavior when the truth is non-monotone far more often than the reverse. We further claim that the asymmetry grows when a problem uses familiar engineering language ("cavitation," "lift loss") compared with the same mathematics under a neutral cover story. That would show a domain prior overriding the stated mechanism, not a failure of arithmetic. If this holds, then final-answer accuracy on physics word problems overstates how well models reason about physics. It also means that the same failure found in code-assisted math, where correct answers can hide unsound logic (Al-Khalili et al.), has a predictable direction in physical domains. Our second claim is narrower. Computing each mechanism in its own context should reduce the asymmetry more than giving the model a code interpreter alone. The basis is the argument of Zhang et al. on divide-and-conquer prompting that isolated sub-contexts stop early errors, and here early priors, from contaminating later steps.

Relation to Prior Work

The closest bridge between LLMs and fluid dynamics is the NL2FOAM work on fine-tuning an LLM to automate OpenFOAM simulations. OpenFOAM is a widely used open-source fluid-flow solver. That work trains a 7B model to write simulation setups that run successfully, but it is restricted to single-phase incompressible flow and judges success by whether the simulation finishes. The obvious extension is to teach such an agent to set up cavitating simulations. We deliberately do not do that. The file gives a reason it would not answer our question: the averaged-turbulence models an agent would configure fail to reproduce the lift bump, so an agent could succeed on every run and still produce physically wrong lift curves. LLM4Fluid repurposes frozen LLMs as time-stepping solvers for 2D flow fields. It involves no symbolic reasoning, so it cannot show whether a model holds a wrong belief about a mechanism. Our proposal differs from both. We measure the model's reasoning about a physical response, and we use the physics as a controlled testbed rather than as the product.

The reasoning literature motivates the measurement design. Yang et al. introduced ConsisEval and the consistency score $CS = P(\text{easy correct} \mid \text{hard correct})$, finding that even GPT-4 Turbo reaches only 92.2%. We borrow their hard/easy pairing. Their pairs, drawn from GSM8K, HumanEval, and IFEval, have no directional physical structure, so they cannot detect a sign-biased prior. Al-Khalili et al. built a judge that reads the structure of generated code (81% accuracy versus 73% for an LLM judge) and showed that on MATH500 execution accuracy barely distinguished sound from unsound logic. We adapt their logic categories to ask whether a model's code actually encodes both mechanisms or only a monotone fit. Cai et al. found that code models plan poorly on their own but improve sharply with expert plans (Starcoderbase-1B Pass@5 from 5.15% to 12.40%). This limits our intervention: if self-generated decomposition is poor, the decomposition step must be checkable, so we score it separately. The intervention itself reuses the divide-and-conquer method of Zhang et al. and competes with multi-tool aggregation (Multi-TAG) and conversational tool use (ChatCoT).

Research Program

CavLift-Probe has three tiers. The recall tier asks qualitative and semi-quantitative questions about the documented Clark-Y behavior: does lift rise before breakdown, roughly where, and why. Ground truth comes from the classic experiments that the LES study validated against (Numachi 1949; Murai 1965). The file notes that exact coefficient tables are not in the summary, so we would digitize them from those sources. This tier is open to memorization and serves as a reference point, not as evidence.

The derivation tier is the core. Each item states a simplified phenomenological lift model and asks the model to derive the lift curve's shape. The model gives lift under partial cavitation as a function of cavity length $\ell$:

$$C_L(\sigma) = C_{L,0}\,[1 + a\,\ell(\sigma)] - b\,\ell(\sigma)^2, \qquad \ell(\sigma) = k\,\max(0,\ \sigma_i - \sigma).$$

Here $C_{L,0}$ is non-cavitating lift, $a$ is the apparent-camber gain per unit cavity length, $b$ sets the lift destroyed by sheet shedding, $k$ sets how fast the cavity grows, and $\sigma_i$ is the inception cavitation number. Lift peaks at $\ell^* = aC_{L,0}/(2b)$, so the curve is non-monotone within the tested range only if $\ell^* < k(\sigma_i - \sigma_{\min})$. This follows from setting the derivative of the quadratic to zero. By randomizing parameters, about half the items have a bump and half decline monotonically. We state plainly that this is a reasoning probe with the structure of the real mechanism, not a physical model of cavitation. Each item appears under three cover stories with identical mathematics: the cavitation framing; a neutral framing (for example, a drug dose with a benefit term and a toxicity term); and an unfamiliar invented-physics framing. The gap in bias between the cavitation and neutral framings is our measure of domain-prior interference.

The consistency tier builds ConsisEval-style pairs. The hard item asks for the full curve and the peak location. The easy item asks only for the sign of the lift change over one interval. We report $CS$ and, more importantly, whether easy-item failures concentrate on intervals where the true sign contradicts the monotone prior.

The intervention works as follows. The model is prompted to list the competing mechanisms, compute each one's contribution over a grid of $\sigma$ in a separate context with a Python tool, and pass only the numeric results to a final combining step. We score the mechanism list against the item's stated terms, so planning failures in the style of Cai et al. can be told apart from combining failures.

Baselines

The naive reference is direct answering with no reasoning scaffold. It sets the floor and shows the raw prior. A second baseline is standard chain-of-thought (the model writes out steps before answering) with self-consistency voting over several samples. This is the cheapest thing a practitioner would try. A third is a long-reasoning model (DeepSeek-R1 class) run with its default thinking budget. This tests whether test-time compute alone erases the bias. The only reported number in the file for such a model in a fluid setting is NL2FOAM's 41.7% accuracy and 22.4% pass@1 for DeepSeek-R1 on OpenFOAM setup, a different task that is not comparable.

The strongest existing competitors are tool-augmented methods. Program-of-thought prompting (writing and running code) with the same Python sandbox isolates the effect of computation. Multi-TAG, which runs chain-of-thought, Python, and a symbolic solver in parallel and selects by consistency, is the strongest published multi-tool method in the file. It reports 6.0–7.5% gains over the strongest baselines on MATH500, AIME, AMC, and OlympiadBench (Multi-TAG), though the file gives no per-dataset absolute scores. The closest method to our intervention is plain divide-and-conquer prompting. It reported GPT-4 F1 of 76.92 versus 69.41 for Tree-of-Thoughts on 5-digit multiplication (Zhang et al.). There is no reported result found on Lacuna for any method on non-monotone physical-response reasoning or on CavLift-Probe.

All methods use the same base models, at least one open 7–8B and one 70B-class model plus one API model. They also share the same sandbox, the same token budget per item (matched by capping samples for voting methods), and the same prompt-tuning budget on a held-out development split. The key ablation runs our decomposition without context isolation: the same mechanism list and code, written sequentially in one context. If isolation matters, the isolated version should beat this ablation on bias asymmetry at matched tokens. Beating the field means cutting the monotone-bias asymmetry by at least half relative to program-of-thought and Multi-TAG without lowering accuracy on the truly monotone control items, which would expose overcorrection.

Why not just…?

Why not just use a bigger reasoning model with chain-of-thought? This is a real possibility, and the benchmark is built to settle it rather than assume the answer. The prior evidence points against scale alone. GPT-4 Turbo still fails easy problems after solving hard ones in 7.8% of cases (Yang et al.), and stronger models showed more unsound code logic on harder problems (Al-Khalili et al.). If frontier reasoning models show no asymmetry and no framing gap, the paper reports that the bias disappears with scale, which is still a useful calibration for smaller deployed models.

Why not just give the model a code interpreter? On the derivation tier this is the most serious threat, because the quadratic can be evaluated exactly. Our bet is that code does not help when the model writes the wrong formula, for example by dropping the camber term or fitting a declining curve, and the code-structure judge will measure exactly that. If program-of-thought matches the decomposition method, we conclude that the bias sits in computation rather than formulation and drop the isolation claim.

Why not just retrieve the experimental data? Retrieval would answer the recall tier, which is why that tier is not our evidence. The derivation and consistency tiers use randomized parameters and invented framings that no document contains. We will also run a retrieval-augmented baseline to confirm it gains on recall and not elsewhere.

Why not just use existing consistency benchmarks? ConsisEval and MATH500 soundness analysis measure whether models are reliable, not whether their errors point in a predictable physical direction. Only items where the monotone answer is sometimes right and sometimes wrong, under matched framings, can separate a directional prior from random noise.

Proposed Experiments

The decisive check is the three-way framing comparison on the derivation tier. For each model and method, we compute the monotone-bias asymmetry: the rate of predicting monotone on true-bump items minus the rate of predicting a bump on true-monotone items. We compare it across cavitation, neutral, and invented framings with the mathematics held fixed. The core claim is supported if the asymmetry is positive and larger under the cavitation framing. It is refuted if the asymmetry is near zero or equal across framings.

Secondary metrics are sign accuracy per $\sigma$ interval, peak-location error relative to $\ell^*$, the hard-to-easy consistency score, and the fraction of generated programs that the adapted structure judge rates as encoding both mechanisms. We will validate that judge against about 200 hand-labeled programs before relying on it. The intervention claim is supported only if isolated decomposition beats the non-isolated ablation and program-of-thought on asymmetry at matched tokens. We will also report the rate at which the model's own mechanism list is wrong, so that gains from better combining are not credited to planning.

As a final external check, recall-tier answers from each method are compared with the digitized Clark-Y lift curve. This tests whether methods that remove the bias on synthetic items also stop describing the real foil as losing lift monotonically.