Many of the phrases that decide the sentiment of a financial sentence are not neat, adjacent word strings. Expressions like cut rates, take a hit, or write down assets get split by inserted words (took a steep, unexpected hit), turned into passives (the assets were written down), or reordered. Language models built for finance learn their phrase knowledge by hiding a run of adjacent tokens during pre-training and asking the model to fill it back in. When the phrase is split, that method sees only unrelated scattered words. We propose to find these expressions through grammatical links rather than word adjacency, and to hide all of their parts together during domain pre-training. The expected result is a model that gives the same sentiment to slashed rates and rates were slashed by the board.
Concretely, we parse a large financial corpus once, offline. We score groups of word stems joined by grammatical links (for example a verb and its object) by how much more often they co-occur than chance. During continued pre-training we then hide every word of a high-scoring group at once, wherever those words sit in the sentence. No parser is needed when the model is used. The claim can fail in a specific way. If the gain is real, it should show up mainly on sentences whose sentiment-bearing expression is split or transformed, and it should hold under the same compute budget as contiguous phrase masking and structure-only masking.
Core Claim
Current finance-specific pre-training teaches phrase knowledge only for adjacent spans. When FLUE Meets FLANG does this by hiding curated financial terms and contiguous multi-token concepts, with an objective that predicts a span from the tokens on either side of it. FinBERT replaces arbitrary contiguous spans. PMI-Masking hides n-grams of length 2 to 5 that co-occur far more often than chance. All three share one blind spot. An expression interrupted by a modifier or reordered by a passive never becomes a masking unit, so the model can rebuild each part from its neighbours. AStitchInLanguageModels calls the resulting tendency compositionality bias: representations stay tied to the literal meaning of each word.
We claim that measuring co-occurrence along grammatical links, not linear order, fixes this blind spot for flexible expressions. We further claim that the fix shows up specifically as sentiment consistency under syntactic transformation of domain expressions. We do not expect large gains on whole standard benchmarks, which are near saturation and contain few split expressions. Our evaluation is built around that expectation.
Research Program
The masking unit is a dependency tuple. A dependency parse is a tree linking each word to the word it grammatically depends on. A tuple is a set of two or three lemmas (dictionary forms of words) connected by such links, together with the relation labels on those links.
For a tuple $t$ with lemmas $w_1,\dots,w_k$, we define the association score as the weakest split of the tuple into two linked parts, following the "weakest link" idea of PMI-Masking:
$$ s(t) = \min_{(A,B)} \log \frac{p(t)}{p(A)\,p(B)}. $$
Here $(A,B)$ ranges over ways of cutting the tuple into two connected sub-tuples. $p(\cdot)$ is the relative frequency of a sub-tuple among all such connected structures in the parsed financial corpus. Taking the minimum means a tuple scores high only if no part of it is merely a frequent word riding along with an unrelated one. We keep a vocabulary of the top-scoring tuples, with the size tuned on held-out perplexity.
At pre-training time we match each sentence's parse against this vocabulary. We hide all subword tokens of a matched tuple jointly, whatever the linear distance between them. The overall masking rate is kept equal to the baselines.
Three prior lines of work shape this design. Syntax-induced pre-training with dependency masking shows that parse trees can be baked into encoder weights during pre-training, giving relation-extraction gains without a parser at test time. We reuse that recipe but change what is hidden: statistically fixed expressions instead of generic arcs. The graph parser of Waszczuk et al. and the analyses of Pasquer et al. and Maldonado et al. establish that split verbal expressions are recovered through tree structure rather than word order, which motivates grammatical links as the unit. SYMPAThy measures how fixed an expression is using entropy over its word forms, word choices and gaps. We use it only for analysis, to sort matched tuples into frozen and flexible groups and to test whether gains concentrate in the flexible group.
On the sentiment side, Wilson et al. showed with hand-built parse features that grammatical position and passive voice change a phrase's contextual polarity. The prior-polarity baseline in that work reached only 48% accuracy on the MPQA opinion corpus. We test whether a neural model can absorb this structure during domain pre-training instead of receiving it as engineered features.
We train BERT-base-sized encoders, starting from the same general checkpoint, with continued pre-training on one financial news and filings corpus. Every masking arm gets an identical token budget and three seeds.
Baselines
The naive reference point is the general checkpoint fine-tuned directly with no domain pre-training. This measures how much any domain exposure buys. The first real baseline is continued pre-training with ordinary random token masking on the same corpus and budget. It isolates the effect of domain data alone.
The strongest existing competitor is contiguous domain phrase masking in the style of FLANG (When FLUE Meets FLANG). We reimplement it on our corpus and budget with the 30/70 financial-term to random-token ratio and the span boundary objective. This is the most direct competitor because it already targets financial multiword concepts, differing from us only in requiring adjacency. For orientation, the paper reports 91.9% accuracy on Financial PhraseBank for FLANG-ELECTRA and 87.2% for FinBERT. Our reimplementation at matched compute is the fair comparison; the published number is a ceiling check on whether our pipeline is competitive at all. We add contiguous PMI-Masking built from the same financial corpus. It is the closest statistical analogue of our score and differs only in linear versus grammatical adjacency.
The key ablation is random dependency-subtree masking. It hides connected subtrees of matched size chosen without any association score, in the spirit of dependency masking. If it matches our method, the benefit comes from syntax generally, not from targeting fixed expressions. A second ablation uses our tuples but ignores relation labels.
Beating the baselines means two things. On the split-and-transformed evaluation described below, the method must exceed both contiguous PMI and random-subtree masking by more than twice the seed standard deviation. On full Financial PhraseBank it must lose no accuracy.
Why not just…?
Why not just use contiguous phrase masking with more steps? The PMI-Masking paper shows that smarter masking can beat much more training, which suggests masking design matters more than extra steps. That evidence concerns contiguous spans on general QA benchmarks, though. It says nothing about split expressions. Our matched-budget comparison answers the question directly, and we also run contiguous PMI at double the budget. If doubling closes the gap on the transformed evaluation, our claim fails.
Why not just mask random dependency subtrees, which already exists? This is a genuine threat. Syntax-aware masking alone might teach constituent binding regardless of which expressions are chosen. The random-subtree ablation settles it. We also check whether our advantage concentrates on tuples that SYMPAThy rates as flexible yet strongly associated. Random subtrees have no reason to favour those.
Why not just augment fine-tuning data with passive and modifier-inserted paraphrases? This may be the strongest cheap alternative, because it targets the evaluation's invariance directly. We run it on top of the FLANG-style model. If augmentation matches our method on the transformed evaluation and on naturally occurring split expressions, pre-training changes are unnecessary and we will say so. We expect augmentation to cover only the transformations it was given. Pre-training should generalise to expressions and variants absent from the fine-tuning data, so we evaluate separately on expressions unseen in fine-tuning. PARSEME 1.2 identifies unseen expressions as the main failure mode, with best open-track unseen F1 of 38.53%.
Why not just run a parser at inference and feed trees to a graph network, or make each expression a single token? The dependency-masking work reports that inference-time parsing adds overhead and error propagation. Single tokens cannot match pull crucial political strings against pull strings, as both AStitchInLanguageModels and Pasquer et al. note. We include a parser-plus-graph-network classifier as a secondary baseline so the cost and accuracy trade-off is measured rather than assumed.
Proposed Experiments
Standard evaluation uses Financial PhraseBank accuracy and FiQA Task 1 sentiment from the FLUE suite. No reported result was found on Lacuna for FiQA aspect sentiment with a comparable metric, so only our reruns will be compared there.
Because whole-benchmark numbers are near saturation, the decisive tests use two targeted evaluations. First, we partition both benchmarks into sentences where a matched sentiment-bearing tuple is split in the surface string and sentences where it is adjacent or absent, and report accuracy per partition. Second, we build a transformation contrast set from Financial PhraseBank sentences containing matched verbal expressions. Each sentence gets variants with inserted modifiers, passivization, clefting and argument fronting, and two annotators verify that the label is unchanged. Following Abeillé, variants whose idiomatic reading is destroyed by the transformation are relabelled rather than discarded. The metrics are accuracy and the rate of consistent predictions across each sentence's variants.
As a check outside finance, we fine-tune on the MAGPIE type-aware split and report sequence accuracy against the 70.47% that DISC reports. This tests whether finance-trained grammatical masking also helps general idiom detection or harms it.
A mechanism probe compares the representation similarity between the parts of an expression in its adjacent and split forms across all masking arms. A parse-noise check re-derives tuples from a hand-corrected sample to confirm that errors from a general-domain parser on financial text do not drive the results.
The proposal is supported if the method wins on the split partition and the contrast set against both contiguous PMI and random-subtree masking at matched compute, with no loss on full Financial PhraseBank. It is refuted if fine-tuning augmentation or random subtrees match it.
References
- When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain
- FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining
- PMI-Masking: Principled masking of correlated spans
- AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models
- Improving Relation Extraction through Syntax-induced Pre-training with Dependency Masking
- A Neural Graph-based Approach to Verbal MWE Identification
- Towards a Variability Measure for Multiword Expressions
- Detection of Verbal Multi-Word Expressions via Conditional Random Fields with Syntactic Dependency Features and Semantic Re-Ranking
- Mapping the constructicon with SYMPAThy. Italian word combinations between fixedness and productivity
- Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis
- Edition 1.2 of the PARSEME Shared Task on Semi-supervised Identification of Verbal Multiword Expressions
- Lexical and Syntactic Rules in a Tree Adjoining Grammar
- Idiomatic Expression Identification using Semantic Compatibility