Visibility-Graph Attention Masks for Language-Model Market Forecasters

Language models are increasingly used to forecast prices. The usual recipe slices a price history into short chunks ("patches") and either writes them out as text or projects them into the model's word-embedding space. Either way, the series becomes a flat line of tokens, and every token is allowed to attend to every other. Code models ran into the same problem years ago. A program is a graph of dependencies, not a string, and models that read it as a string learned surface shortcuts. The fix that worked was to extract the dependency graph and restrict attention to follow it. This note proposes the analogous fix for markets. We build a "visibility graph" over the patches of each input window, linking two price extremes when a straight line between them clears every bar in between, which roughly captures a support or resistance level revisited after a long gap. We then mask a subset of attention heads so they can attend only to local neighbours and to these graph edges.

In plain terms, the claim is that telling a forecaster which distant moments are geometrically related improves cross-sectional prediction and risk-adjusted returns more than handing it the same graph as extra input features. The benefit should be concentrated in volatile, regime-shifting windows. If the claim holds, it shows that the structural-loss lesson from code transfers to continuous, noisy sequences. If it fails, it tells practitioners that flat attention already recovers this structure, and that the reported successes of graph preprocessing come from the features rather than from the constraint on attention.

Core Claim

GraphCodeBERT showed something specific. It masked attention so that variable tokens attend only to their data-flow sources and neighbours. That beat unconstrained attention on code search (MRR 0.713 vs. CodeBERT's 0.693), and attention analysis showed heads preferring data-flow edges over sequential order. Work on vulnerability detection by Chakraborty et al. and Zhang et al. motivates why this matters: flat sequence models latch onto superficial tokens rather than logic.

On the market side, Price graphs turned price series into visibility graphs and used node centrality to discount isolated spikes, reporting a 47.91% cumulative return on CSI-300 in 2019. However, it used the graph only as static, non-differentiable preprocessing feeding an RNN, never as a constraint on attention. Language-backbone forecasters such as Time-LLM and VTA treat patches as a fully connected flat sequence. DishFT-GNN uses graphs, but predefined graphs between stocks, not structure extracted from each price path.

The new element is to compute a visibility graph from the input window and enforce it as a structural attention mask inside a domain-specialized, compact language backbone. The comparison that matters is mask versus the same graph supplied as features.

Research Program

For a lookback window split into $n$ patches, we summarise each patch $a$ by its high $h_a$ and, separately, its low. Patches $a<c$ are connected in the natural visibility graph if every intermediate patch lies below the line joining them:

$$h_b < h_c + (h_a - h_c)\frac{c-b}{c-a} \quad \text{for all } a<b<c,$$

with the mirrored condition on lows. The edge set $E$ is the union of both graphs. It is computed only from the lookback window, so it cannot leak future information, and it costs $O(n\log n)$ per window.

In a designated half of the attention heads, the logit between query token $i$ and key token $j$ becomes

$$\frac{q_i^\top k_j}{\sqrt{d}} + M_{ij}, \qquad M_{ij} = \begin{cases} 0 & |i-j|\le w,\ (i,j)\in E,\ \text{or } i \text{ is a prompt token} \\ -\infty & \text{otherwise.} \end{cases}$$

Here $q_i, k_j$ are query and key vectors, $d$ is the head dimension, and $w$ is a small local window. The other heads stay unrestricted. This mirrors GraphCodeBERT, where comment tokens see everything and only structure tokens are constrained. The natural-language prefix, built as in Time-LLM's prompt-as-prefix with asset, frequency and summary statistics, remains fully visible, so verbal context is never cut off.

The backbone is a GPT-2-scale language model with Time-LLM-style patch reprogramming. We specialize it by fine-tuning on the training splits of financial series, following the domain-specialization evidence that compact focused models beat large generalists in finance (FinVerse finds a small model, Reverso Small, ranks best overall among 43). We deliberately unfreeze the backbone with low-rank adapters, because imposing a mask on a frozen model alters computation it was never trained for. We also apply the identical mask to a plain numerical patch Transformer. This tells us whether any gain belongs to the language interface or to structured attention generally.

Baselines

The naive reference points are two. The first is a no-change forecast (last value or zero return), which anchors information coefficient and Sharpe at chance. The second is the same specialized backbone with full, unmasked attention. That second model is the most important comparison, because it differs from the proposal only in the mask. Beating it means a statistically significant gain in rank information coefficient and Sharpe across walk-forward folds, with five seeds and identical data, adapter rank, step count and hyperparameter search budget.

The strongest language-backbone competitor is VTA. It trains a Qwen2.5-7B reasoner with an inverse-MSE reward to guide a GPT-2 forecaster, and reports aggregate MSE 0.1178 (runner-up CALF 0.1235) and a Markowitz Sharpe of 1.7190 on ACL18 StockNet plus global indices. We will evaluate on the same splits and lookback. Matching VTA's MSE with a model lacking a 7B reasoner would already be meaningful. Beating its Sharpe at equal cost assumptions would count as a clear win.

The strongest numerical competitor is WaveLSFormer, a learnable-wavelet Transformer trained directly on trading objectives. On hourly U.S. equities it reports a Sharpe of 2.157 ± 0.166, against 1.024 ± 0.122 for a plain Transformer. Because its gain comes largely from its trading-objective loss, we will compare like with like. We add the visibility mask to WaveLSFormer's own training objective and ask whether the mask raises it beyond the reported band on the same 0.7/0.1/0.2 temporal split.

FinCast reports 20% lower MSE than general foundation models zero-shot. We will include it where checkpoints are available. On FinVerse, the file lists leaders by rank only: Reverso Small overall, TiRex 1.1 in point accuracy, Chronos-2 Synth in portfolio backtests. I found no reported result on Lacuna giving their numeric scores, so we will rerun them under FinVerse's protocol. The Price graphs pipeline is reproduced on CSI-300, where its 47.91% return is the target.

Three ablations isolate the key ingredient. The first is a random mask with the same sparsity and degree distribution, which rules out "sparsity as regularization." The second keeps full attention but adds visibility-graph features (node degree, collective influence) as extra input channels. The third uses a learned soft bias on edges instead of a hard mask.

Why not just…?

Why not just feed the visibility-graph features as inputs and keep full attention? This is the most serious threat. Price graphs succeeded using graph features without any attention constraint, and a Transformer given degree and centrality per patch might do as well. Nothing in the file settles it for time series. GraphCodeBERT's evidence favouring masking comes from code, where dependencies are exact rather than noisy. The features-as-input ablation is therefore a primary experiment, not a footnote. If it matches the mask, we will report that the useful content is the graph statistics, and the attention constraint is unnecessary.

Why not just use a compact numerical model and skip the language backbone? Purely numerical WaveLSFormer reaches a reported Sharpe of 2.157, and Patch-wise Structural Loss improves DLinear by 5.22% MSE without any language model. We take this seriously by applying the mask to a numerical Transformer as well. The core question, whether structural masking helps, is answered on both backbones. The language backbone's advertised value lies in few-shot and cross-asset transfer (Time-LLM reports over 14% zero-shot error reduction) and in the fully visible verbal prefix. We therefore test it specifically in low-data and held-out-asset settings rather than assume it.

Why not just let full attention learn the structure, given enough data? GraphCodeBERT found that unconstrained heads partly rediscover data-flow edges. Markets may behave the same way. We measure the fraction of attention mass that full-attention heads already place on visibility edges, relative to their share of all pairs. If that fraction is high and the mask adds nothing, flat attention suffices. If it is near the chance level and the mask helps, the inductive bias is doing work that data alone does not.

Proposed Experiments

We evaluate on four settings. The first is FinVerse, using its information-coefficient, hit-ratio, Sharpe and maximum-drawdown families on daily series. The second is ACL18 StockNet with the 2024–2025 Dow Jones, FTSE China A50 and EURO STOXX 50 indices, for the head-to-head with VTA. The third is the hourly U.S. equity set, for WaveLSFormer. The fourth is CSI-300 2010–2019, for the Price graphs comparison. All portfolio results are reported at zero and at several proportional transaction-cost levels, since the strongest prior trading results ignore frictions. Because FinVerse finds point accuracy correlates with portfolio performance at only $r=0.37$, the headline metrics are information coefficient and after-cost Sharpe, with MSE secondary.

The decisive checks are four. First, the masked model must beat both the full-attention and random-mask controls. Without that, there is no effect of structure. Second, it must beat the features-as-input ablation. Without that, the mask is not the mechanism. Third, the gain should grow in the top volatility tercile of test windows and with longer lookbacks, where visibility edges span many patches. On smooth generic data (ETT, Weather) the gain should shrink toward zero. Finding the opposite would mean the mask acts as a generic regularizer rather than capturing market geometry. Fourth, the gain should persist after transaction costs.