Terms of Art as Domain Idioms: Teaching Models to Notice Sense Switches

In professional text, ordinary words often carry a fixed technical meaning. In a contract, "consideration" means the value each party exchanges, not thoughtfulness. "Discovery" is a pretrial procedure, and "good faith" is a legal standard of conduct. Medicine and cloud operations have their own versions, such as a "dry run" or a "cascade." For a language model these phrases are idioms that run in reverse. On web text the everyday sense dominates, so a model reading a legal document is pulled toward the wrong meaning. The domain sense is rare and does not follow from the parts of the phrase. Work on idioms has measured this pull and fought it, but only for colloquial English expressions like "kick the bucket." Work on domain reasoning has documented expert-task failures without asking whether some of them begin as a misread word.

This note proposes treating terms of art as domain-specific non-compositional expressions, meaning phrases whose meaning cannot be built from their words. The central question is whether recognizing that a familiar phrase has switched into its technical sense is a skill that transfers to terms never seen in training, or whether it must be memorized one term at a time. Both existing idiom methods and retrieval methods assume the second answer and fail on unseen expressions. If the first answer holds, a small, cheap training signal could close part of the domain gap for open models. Those models currently sit far below closed ones on expert benchmarks.

Core Claim

Training an open 7–8B model on contrastive minimal pairs will improve its sense judgments on held-out terms and in held-out domains. A minimal pair here is the same phrase used once in its everyday sense and once in its technical sense, with the model asked to explain which meaning applies. The prediction is that this beats two alternatives given the same examples without the pairing: ordinary fine-tuning and loss upweighting. If it does, the gain comes from learning to detect sense switches, not from extra exposure to the domain.

A secondary claim is that some downstream legal-reasoning errors are caused by this misreading. Fixing it should measurably help exactly those items.

Grounding in Prior Work

The idea draws on three lines of work.

Evidence of the gap. Two results motivate the proposal:

  • GPT-3 failed "nearly 50% of the time" on a 75-question idiom set, reading figurative phrases literally, for example treating "all at sea" as a sign of progress on a technical task (Getting BART to Ride the Idiomatic Train).
  • LegalBench shows that models can state a legal rule yet stumble when applying it to new facts.

Neither paper asks whether the application failure partly starts at the word level. That is the question tested here.

Methods that are reused. - Getting BART to Ride the Idiomatic Train introduced GIEA, an adapter trained to pull idiom representations toward embeddings of their dictionary definitions. - Unified Representation for Non-compositional and Compositional Expressions added PIER+, which blends a literal model and an idiom model layer by layer. It also documented "semantic blurring": literal and idiomatic uses of the same phrase receive nearly identical representations.

The definition-anchoring idea and the blurring measurement are borrowed from these papers. Their limits are explicit in their own results: they cover only colloquial idioms from the MAGPIE dataset, and they cannot interpret unseen expressions because they depend on a precompiled dictionary. This proposal differs on both points. Its target is domain vocabulary, and its central test is generalization to terms held out from training.

Results that constrain the design. - Crossing the Threshold found that translation models stay literal until an idiomatic sense reaches roughly 10% of the data. Simply multiplying the training loss on idiom sentences yields 10.4–13% gains in idiomatic accuracy for French and Finnish. This is the simplest competitor and must be beaten on equal data. The same paper reports that both upweighting and retrieval fail on idioms not seen in training, which is where this proposal looks for its advantage. - TheoremQA found that pasting theorem statements into the prompt "did not provide a significant boost." That is suggestive evidence that having a definition is not the same as applying it. It concerns theorems rather than word senses, though, so it is a hypothesis to test, not a settled answer. - Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining shows that legal pretraining objectives designed around legal language beat plain masked-word prediction. It covers contract templates and clause types, and it leaves terms of art untouched. - Investigating Linear Probe Robustness found that medical "truth directions" survive changes in writing style but drop 0.12–0.21 AUROC when moved to a different dataset. This is a warning that representation-level gains can be artifacts of one dataset, so every probe analysis here is evaluated across corpora.

Research Program

Benchmark: TermShift. The first artifact is a new benchmark built in three domains:

  • Law: contracts and filings from the LEDGAR, SEC and CUAD collections used in the legal-pretraining paper.
  • Medicine: text in the style of the MedQA, MedMCQA and MedRedQA suite.
  • Cloud and IT: text in the style of TechQA support threads.

Construction proceeds as follows. Annotators compile a few hundred candidate terms from professional glossaries. For each term, sentences are mined from the domain corpus and from FineWeb (The FineWeb Datasets), so that each term appears in both senses. An LLM pre-labels the sense, and two humans verify every test item. Each item has three parts: a sentence, a two-way sense question, and a short free-text gloss. Terms are split so that test terms never appear in training. One domain, cloud and IT, is held out entirely.

Training method: paired sense contrast. The method is applied as a low-rank adapter, a small set of trainable weights added to a frozen model, on Llama-3-8B and Qwen2.5-7B. Its output target is generative: for each sentence the model writes which sense is active and why. Its representation target borrows GIEA's definition anchoring and adds pairing. The two senses of a phrase must be pushed apart, and each must be pulled toward its own definition:

$$\mathcal{L} = \mathcal{L}_{\text{gen}} + \lambda \left[ m - \cos(h_t, d_t) + \cos(h_t, d_o) \right]_+$$

The terms are defined as follows:

  • $\mathcal{L}_{\text{gen}}$ is the cross-entropy loss on the sense explanation.
  • $h_t$ is the model's pooled hidden state for the phrase in a technical-sense sentence.
  • $d_t$ and $d_o$ are fixed sentence embeddings of the technical and ordinary definitions.
  • $m$ is a margin, $[\cdot]_+$ keeps only positive values, and $\lambda$ weights the two losses.

The same loss is applied symmetrically to ordinary-sense sentences.

The key ingredient is the pairing. Each technical-sense sentence is shown alongside a matched ordinary-sense sentence for the same phrase, so the model must find the contextual cue that separates them. Memorizing that the phrase "usually means X in contracts" will not solve the pair.

Diagnostic analysis. Semantic blurring is measured as the cosine distance between technical and ordinary uses of the same phrase, following the PIER+ paper. A linear probe tests whether the sense distinction is already present inside base models whose outputs get it wrong. The probe is trained on legal text and evaluated on medical text, so that the corpus-shift failure documented in the medical probing paper is checked directly rather than assumed away.

Baselines

Do-nothing reference. The untuned base model answers the TermShift sense questions zero-shot and with a few examples in the prompt. It anchors every comparison. It also tells us how large the gap is for open models, which is not yet known: no reported result was found on Lacuna for term-of-art sense disambiguation in any domain.

Plain fine-tuning on the same data. The same sentences and the same sense labels are used for ordinary supervised fine-tuning. The pairing is removed and so is the representation term. This is the ablation that isolates the key ingredient. It uses an identical number of examples, the same adapter size, the same training steps and the same hyperparameter-search budget. The proposal counts as beating it only if paired contrast wins on held-out terms by a margin larger than the spread across three random seeds.

Loss upweighting. This baseline follows Crossing the Threshold: technical-sense sentences are mixed into domain text, and their loss is multiplied so their effective density exceeds the roughly 10% tipping point. It is the strongest published method for rare non-compositional senses. That paper reports 10.4–13% absolute idiomatic-accuracy gains in translation. Those gains were measured on expressions seen in training, and the paper says the method does not reach unseen idioms. That pattern is exactly what should hold here if the proposal is right. To be fair, the multiplier is tuned with the same search budget, and the baseline sees the same sentences.

GIEA-style anchoring without pairing. This keeps the definition-similarity term but applies it to unpaired sentences, testing whether the pairing or the anchoring does the work. Following the PIER+ paper, its blurring metric is reported alongside accuracy. That paper reports a +0.49 cosine-distance separation over GIEA for colloquial idioms; that figure is a reference point, not a target for this domain.

Glossary in the prompt. Both the ordinary and the technical definitions are retrieved and placed in the prompt. Supplying only the technical definition would presuppose the answer, so both are given.

Frontier ceiling. A large closed model is run zero-shot as a ceiling reference rather than a competitor.

Downstream references. For LegalBench, published GPT-3 (text-davinci-002) numbers anchor the scale:

  • Abercrombie: F1 0.42 few-shot and 0.54 with chain-of-thought.
  • Diversity Jurisdiction: F1 0.74 with chain-of-thought.

All of these are reported in LegalBench. The comparison is between the same base model before and after each intervention, not against those older numbers directly.

Why not just…?

Why not just put the definition in the prompt? This is the most serious threat. Terms of art, unlike theorems, may need only a definition, so the TheoremQA null result may not carry over. The glossary baseline therefore receives both senses and must still choose between them, which isolates sense selection from missing knowledge. If glossary prompting matches paired training on held-out terms, the bottleneck is missing knowledge rather than noticing a switch. The training method would then be unnecessary, and that conclusion would be reported. Glossary prompting also requires knowing in advance which phrases are terms of art. The held-out cloud domain tests performance when no glossary exists.

Why not just upweight domain sentences, as in idiom translation? Upweighting clearly works on terms present in the data, and the proposal expects it to match paired training on training terms. The claim stands or falls on held-out terms and the held-out domain. That is where Crossing the Threshold itself reports that its methods fall short. If upweighting also transfers, the lesson is "density matters, not contrast," and the pairing ingredient is dropped.

Why not just use a bigger model? Frontier models may already know common legal terms. However, GPT-3's roughly 50% literal-reading failure on idioms (Getting BART to Ride the Idiomatic Train), and open models scoring below 15% on TheoremQA, suggest the gap is real in the open-model regime. That regime is the one targeted here. If the frontier model sits near ceiling on TermShift, the contribution narrows to cheap adaptation for small models, and the note will say so.

Proposed Experiments and Decisive Checks

The first and decisive test is transfer to held-out terms on TermShift. The metrics are:

  • sense accuracy;
  • gloss correctness, judged by humans on a sample;
  • blurring distance.

Results are reported for each domain and separately for the fully held-out cloud domain. Paired contrast must beat plain fine-tuning and upweighting on unseen terms while matching them on seen terms. Matching on seen terms with no advantage on unseen terms would falsify the core claim.

The second test is a no-harm check. The ordinary-sense half of TermShift must not degrade, and the model is evaluated on MAGPIE sense classification and span detection. The PIER+ paper reports general-language understanding held within ±1% of baseline under its intervention. A comparable bound is required here, so that the adapter does not learn to over-read every phrase as technical.

The third test asks whether fixing word-level misreading helps downstream reasoning, using LegalBench tasks whose inputs contain a TermShift term, such as Hearsay, CUAD clauses and Abercrombie. Items are divided into those where the base model misreads the key term in a probing question and those where it does not. Gains should concentrate in the misread subset. If the overall gain is spread evenly, or if no gain appears, the proposed causal path from misreading to reasoning error is not supported.

The fourth check concerns the probe analysis. The internal sense direction must work across corpora, trained on legal text and tested on medical text, before any representation-level claim is made. This avoids the dataset-artifact failure documented in Investigating Linear Probe Robustness.