Do Language Models Compute Ergonomic Risk or Recall It?

Language models are starting to answer workplace-safety questions. People ask them whether a lift is dangerous, which workstation change will reduce injury, or what a worker's complaint implies. These questions have exact answers when the numbers are known. The Revised NIOSH Lifting Equation and posture-scoring tables such as REBA (Rapid Entire Body Assessment) turn load mass, reach distance, lift height, frequency, and joint angles into a risk score by deterministic rules. The web, however, is full of safety slogans: "lift with your legs," "sitting is safer," "construction is high-risk." A model can score well on static ergonomics questions just by matching these slogans. When the slogan and the physics disagree, it will be wrong.

We propose to build a generator of ergonomic reasoning problems in a symbolic space. Each problem is a set of task parameters whose risk label is computed exactly by the lifting equation or scoring table. An LLM then renders that set of parameters into natural-language workplace vignettes. Crucially, the surface story is varied independently of the underlying parameters. That lets us write the same instance so that its surface cues either agree or conflict with its true risk: a dense small part that is actually over the limit, a bulky box that is actually light, a "modern seated assembly" job whose neck and shoulder load is the dominant risk. The gap in accuracy between cue-agreeing and cue-conflicting renderings of identical instances measures how much a model relies on recalled associations rather than computation. That is a contamination-style question asked in a domain where no verifiable dynamic benchmark exists. The same generator then serves as a source of verified training data, and we test whether it reduces that reliance better than natural-language augmentation does.

Core Claim

Our claim has three parts. Static or cliché-consistent ergonomic questions overstate how well language models reason about occupational risk. The overstatement shows up as a measurable drop when surface cues are made to conflict with the verified label while the underlying instance stays fixed. Fine-tuning on symbolically generated, verifier-labeled cases shrinks this drop more than paraphrase or natural-language mutation of the same seeds.

The new element is not dynamic generation itself. DyVal generates reasoning problems from directed acyclic graphs, with controllable depth, and shows that models strong on static benchmarks collapse on its problems. Its domain is arithmetic and formal logic. Adaptive Problem Generation via Symbolic Representations edits math word problems in SymPy code and verifies them with solvers. It reports pairwise embedding diversity of 0.37 for symbolic edits versus 0.07 for natural-language edits, and an 8% versus 4% accuracy gain from 100 seeds. Its domain is algebra, and its seeds may themselves be contaminated. We borrow both paper's methods: the separation of structure from description, and verification by executable rules.

What we add is a physically grounded rule system from occupational health. We also add an explicit design that controls surface cues separately from structure. Neither prior paper does this, because in math there is no strong prior "safety slogan" to conflict with.

Research Program

The symbolic space. It has three task families, each with an exact executable scorer.

The first family is single lifts scored by the Revised NIOSH Lifting Equation. The recommended weight limit is a load constant multiplied by factors for horizontal distance, vertical height, travel distance, trunk twisting, lift frequency, and grip quality. The risk label is the lifting index, defined as load divided by that limit, binned at 1 and 3. Parameter ranges are anchored to a real laboratory protocol so the cases stay realistic: lifts from mid-shin, knuckle, and shoulder height with loads of 2.7 to 13.6 kg, as in the lifting dataset of Towards video-based injury risk assessment.

The second family is composite shifts made of several lifting subtasks. The number of subtasks and their order act as a depth knob, analogous to DyVal's graph depth.

The third family is workstation-redesign comparisons scored with REBA/RULA region subscores. The model must say which redesign lowers the maximum region risk. Here we encode the epidemiological finding that interventions relocate strain rather than remove it. The electronics manufacturing study of Feng Yang et al. found that seated work reduces lower-limb pain but raises neck and shoulder risk. Yin et al. found multi-site symptoms dominate. Industries and body sites for the vignettes are drawn from the prevalence studies on shoulder disorders across 15 industries and wrist postural injury. These studies motivate the task design; they do not supply the method.

Rendering and verification. An LLM turns each parameter set into a vignette. Every rendering is then checked twice before use:

  1. A separate extractor model and a regular-expression check recover the parameters from the text. Any rendering whose recovered parameters do not match the source is discarded. This is the analogue of solver filtering in MILP-Evolve.
  2. A cue tag records whether the surface story (industry, object description, posture wording) points toward or away from the verified label.

We also generate counterfactual pairs that change exactly one factor, such as moving the load 15 cm farther from the body. For these pairs, the correct direction of change in risk is known from the monotonicity of the equation.

Diagnostics and training. The main diagnostic is the cue-conflict gap: accuracy on cue-agreeing renderings minus accuracy on cue-conflicting renderings of the same instances. We also report counterfactual direction consistency and accuracy as a function of composite depth. On the training side, we fine-tune open-weight models of roughly 1.5B to 8B parameters on generated cases. Evaluation always uses parameter regions and task templates that were held out from training.

Baselines

Naive reference points. The first is the majority-class label for each family. The second is a "slogan scorer": a keyword rule that assigns risk from industry and object words alone. On cue-conflicting items this scorer should be at or below chance by construction, which confirms the conflict manipulation works. The third is zero-shot prompting of each evaluated model with chain-of-thought.

Strongest existing method in the domain. This is ErgoChat, a MiniGPT-v2 model fine-tuned to answer REBA-based risk questions about construction images. It reports 96.5% question-answering accuracy on its own static test split of 1,900 image-text pairs. It is the only language-model ergonomic assessor in our file, and its static evaluation is exactly what we suspect of overstating ability.

Our benchmark is textual, so we cannot run ErgoChat on it directly. For a fair comparison we instead use the same backbone family (LLaMA2-7B) fine-tuned on text converted from ErgoChat-style REBA question-answer pairs, with an equal example count and tuning budget. We report the cue-conflict gap for this model.

No reported result was found on Lacuna for any language model on text-based ergonomic risk reasoning. There is also no published lifting-equation benchmark, so all numbers on our suite will be new. Beating this baseline means a cue-conflict gap at least halved relative to it, with no loss on cue-agreeing items.

Strongest augmentation competitor. This is natural-language mutation: the LLM directly rewrites seed vignettes into new ones, which is the baseline arm in Yeo et al. We also include paraphrase-only augmentation. Both use the same seeds, the same number of training examples, the same generator model, and the same fine-tuning compute. For the natural-language arm we compute labels with the verifier where parameters can be extracted and drop the rest, so both arms train on correct labels. Any remaining difference is then about structural diversity, not label noise.

Tool-augmented pipeline. The model extracts parameters and calls the exact scorer. This separates extraction errors from errors in applying the rule.

Key-ingredient ablation. We train on symbolic data rendered with only cue-agreeing stories. This isolates whether the cue-decorrelated rendering, rather than the symbolic generation alone, drives any reduction in the gap.

Why not just…?

Why not just use the deterministic lifting equation and REBA tables, with no language model at all? When measured parameters are available, this is correct, and we use exactly these tables as our verifier. But assistants are already asked to reason from narrative descriptions where no one has entered numbers. The question we study is whether they do so faithfully. The tool-augmented baseline is the practical version of this alternative. If it closes the cue-conflict gap, the finding is that ergonomic assistants should be built as extract-then-compute systems. The benchmark would then become a test of extraction robustness under misleading stories, which is still a useful and publishable outcome. We report extraction accuracy separately so this outcome is visible rather than hidden.

Why not just paraphrase existing ergonomics questions to defeat memorization? Rethinking Benchmark and Contamination with Rephrased Samples shows that rephrased test items still carry the original's structure and slip past detection. Yeo et al. measure natural-language rewriting at 0.07 embedding diversity versus 0.37 for symbolic edits. Paraphrase also cannot set the true label against the surface cue, because it never touches the parameters. The paraphrase augmentation arm tests this directly.

Why not just run membership inference such as Min-K% Prob on public ergonomics material? Chi et al. used it to show older causal benchmarks leaked. But On the Fragility of Benchmark Contamination Detection and Data Contamination Can Cross Language Barriers show such detectors fail under rewording and translation. They also need token probabilities that closed models do not expose. More importantly, the harm we care about is not verbatim memorization of a test set but reliance on memorized associations, which no membership test measures. We run Min-K% Prob on public guidance text as a secondary diagnostic only. We check whether its scores correlate with the cue-conflict gap across open models; if they do not, that illustrates why a behavioral measure is needed.

Why not just collect a private, expert-annotated video set? Such a set would be valuable, but it is costly: ErgoChat's reached 1,900 pairs. It is static, so it cannot vary difficulty or decorrelate cues. And it leaks once released. We treat a small expert-written vignette set as a transfer check, not as the benchmark.

Proposed Experiments

Experiment 1: evaluation. We evaluate open and closed models zero-shot on all three families. The first decisive check is the result pattern. Cue-agreeing accuracy should be high. Cue-conflicting accuracy on identical instances should be much lower. Accuracy should fall with composite depth, in the way DyVal reports a 23% drop for GPT-3.5-Turbo from its easiest to hardest arithmetic level. Counterfactual direction consistency should lag single-item accuracy. If the cue-conflict gap is near zero for current models, the core hypothesis is wrong and we would report the suite as a stress test rather than a finding.

Experiment 2: training comparison. We compare symbolic-cue-decorrelated training against natural-language mutation, paraphrase, and the cue-agreeing-only ablation. Seeds, example counts, and compute are equal across arms. The decisive check is whether the symbolic-decorrelated arm reduces the gap on held-out parameter regions and templates, and whether the cue-agreeing-only arm does not. That result would locate the benefit in decorrelation rather than in volume.

Experiment 3: transfer to human-written text and real lifts. We test on a small set of vignettes written by occupational-health practitioners, who are blind to our generator. These cover industries named in the prevalence literature. We also test on text descriptions of the recorded lifting conditions of Zhu, Mu, Radwin and Li's lifting dataset. Their pose-only Transformer reaches 74.8% on light-versus-heavy classification and 50.8% on three-way classification, figures we cite only as context for how hard intermediate loads are, not as a comparable score.

If gains on generated data do not carry over to the practitioner set, we will conclude that the generator teaches the template rather than the physics. That is the most important way this proposal could fail, and the experiment is designed to show it plainly.