Letting the Image Change the Grammar: View-Conditioned Low-Rank Transitions for Multimodal CRF Tagging

When a model tags names in a tweet that comes with a photo, the usual recipe is to merge the photo's features into every word's feature vector and then hand the result to a conditional random field (CRF). A CRF is an output layer that scores whole label sequences instead of one word at a time, so it can forbid invalid sequences such as an "inside-a-person-name" tag that follows an "outside" tag. Merging features this way repeats the familiar cost of concatenating views. The input dimension grows, the model overfits, and a noisy or irrelevant image can drown out the text. This proposal moves the second view somewhere else. The image does not touch the per-word scores. Instead it adjusts the CRF's transition table, the small matrix that says how plausible each label-to-label step is, through a low-rank, gated correction that costs a few thousand parameters and leaves exact inference unchanged.

The motivation is a mismatch in where the information lives. A post-level image mostly tells you which kinds of entity are likely and how spans should hang together. A photo of a stadium raises the plausibility of an ORG span continuing across "Manchester United". It rarely tells you anything specific about token seven. Current multimodal taggers spend their engineering on encoder-side fusion and then decode with a static transition matrix. Placing the view inside the pairwise structure of the CRF is a cheaper, more interpretable channel that can be switched off when the image is irrelevant. It has not been tested against encoder-side fusion.

Core Claim

For multimodal and multi-view sequence labeling, a secondary view brings more benefit per parameter, and more robustness to irrelevant inputs, when it conditions the CRF's transition potentials through a low-rank gated correction than when it is fused into per-token emission vectors. This should hold most strongly when labeled data are scarce and the view is noisy. Concretely, with text emission scores $E(y_t;\mathbf{x},t)$, a base transition matrix $T$, and a pooled view vector $z$ (for example an image embedding), the sequence score is

$$\text{Score}(\mathbf{x},z,\mathbf{y})=\sum_{t=1}^{T_{\text{len}}} E(y_t;\mathbf{x},t)+\sum_{t=1}^{T_{\text{len}}-1}\Big[T(y_t,y_{t+1})+g(z)\,a_{y_t}^{\top}\,\mathrm{diag}(\phi(z))\,b_{y_{t+1}}\Big].$$

Here $a_y, b_y \in \mathbb{R}^r$ are learned rank-$r$ label embeddings for the "from" and "to" side of a transition, $\phi(z)\in\mathbb{R}^r$ is a small projection of the view, and $g(z)\in[0,1]$ is a scalar relevance gate. Because the correction is one $|\mathcal{Y}|\times|\mathcal{Y}|$ matrix per sequence, Viterbi decoding and forward-algorithm training are exactly as before. The cost is roughly $d_z r + 2|\mathcal{Y}|r$ parameters, where $d_z$ is the view dimension. Concatenating $z$ into every token adds $d_z h$ parameters to the first projection, where $h$ is the hidden size.

Relation to Prior Work

Two recent multimodal entity recognition systems motivate the gap. MPCFN filters and cross-attends image regions with BERT tokens, then decodes with a standard linear-chain CRF. It needs an ad hoc "flooding" floor on the training loss to stop the fused representation from overfitting. MAHE uses mixture-of-experts text routing and a multi-scale vision state-space model before a CRF with static transitions. Its authors report that it remains distracted by task-irrelevant images. Both put all fusion in the encoder. We leave the encoder text-only, or keep theirs unchanged, and test the decoder as the place where the views meet.

Two lines of work supply the method. Jagannatha and Yu showed that transition scores conditioned on token context reduce false boundaries in clinical text, reaching 0.8266 strict precision. Their potentials are full-rank, come from a CNN over concatenated word and character features, and carry substantial parameter overhead. We keep the idea of context-dependent transitions but make them low-rank and condition them on a different view, not on the same text. Chandra et al. factor dense pairwise affinities in continuous Gaussian CRFs as inner products of low-dimensional embeddings. We borrow the factoring trick for discrete label-to-label potentials. There the factorization is over pixels for speed; here it is over labels, for sample efficiency and view control.

The concatenation literature constrains the design. MLLE shows that concatenated multi-view descriptors become sparse and lose meaningful distances: PCA on the concatenation reached an $A_z$ of 0.526 against 0.990 for view-preserving alignment. scMFF found that a plain weighted sum beat attention, mixture-of-experts and transformer concatenation on 41 of 42 single-cell datasets. It also found that multiplying representations from different mathematical spaces destroyed signal. We take the second finding seriously: the gated correction never multiplies image features with text features. It only adds a bounded term to label-pair scores.

Research Program

We will build the decoder as a drop-in CRF layer and test it in two regimes. The first is image-and-text entity recognition on Twitter-2015, Twitter-2017 and WuKong. There $z$ is a pooled frozen image embedding, and in a second variant a per-position $z_t$ obtained by letting each token attend over image regions. The second regime contains no image at all. It is a small-data, text-only multi-view setting on the tmVar genomic variant corpus (4,783 sentences), where the views are word embeddings, character-CNN features and a frozen contextual encoder. In the standard design these are concatenated, as in DeepVar. Here we move the auxiliary views into the transition correction. This tests whether the effect is about where views enter a CRF, or something specific to images.

The analysis has three parts. The first is sample efficiency: we subsample training data to 10%, 25%, 50% and 100%, and predict that the gap over concatenation widens as data shrink, since emission-side fusion adds capacity that grows with $d_z h$. The second is robustness: we replace each test image with one from a random other post, and also with blank images. The prediction is that our model falls back toward its text-only score, because $g(z)$ shrinks and $T$ remains, whereas emission fusion degrades below text-only. The third is interpretability: we inspect the learned rank-$r$ corrections to see whether they encode entity-type priors, for example raising B-PER→I-PER when faces are present.

Baselines

The naive reference point is a text-only BERT-CRF with a static transition matrix, trained with the same encoder, optimizer, and tuning budget of an equal number of hyperparameter trials. Every multimodal gain is measured against it. Beating it is necessary but not sufficient.

The concatenation baseline appends $z$ to every token representation before the emission layer, with dropout and weight decay tuned over the same budget. This is the configuration the curse-of-dimensionality argument predicts will overfit. We count it as beaten if our model is better on F1 at full data and clearly better at the 10% and 25% subsamples, averaged over five seeds with paired significance tests.

The weighted-sum baseline follows scMFF. It projects each view to a common size and takes a learned convex combination before the emissions. It is the strongest low-capacity competitor, and the one most likely to match us in small-data settings.

The strongest existing method is MPCFN. It reports 76.74% F1 on Twitter-2015, 87.61% on Twitter-2017 and 82.35% on WuKong (MPCFN); MAHE reports 75.34 and 87.41 on the two Twitter sets (MAHE). Published numbers may reflect different backbones and tuning. We will therefore reimplement MPCFN's encoder under our compute budget and run it two ways: with its standard CRF, and with our transition correction added on top. The question is not only whether a text-only encoder plus our decoder approaches MPCFN at a fraction of the parameters. It is also whether our decoder adds anything on top of MPCFN's own fusion. Matching MPCFN within one F1 point with a text-only encoder, or improving MPCFN when stacked, would count as a positive result. On tmVar, the reference is DeepVar's 90.78% F1, against roughly 89% for a vanilla BiLSTM-CRF.

Three ablations isolate the key ingredient. The first, and most important, feeds the same $\phi(z)$ into the emissions as a rank-$r$ label bias with an identical parameter count, so that placement is the only difference. The second uses a static low-rank correction with no dependence on $z$, which tests whether gains come from conditioning or merely from extra transition capacity. The third uses full-rank $z$-conditioned transitions in the style of Jagannatha and Yu, which tests whether the low rank matters. We will also remove the gate $g$.

Why not just…?

Why not just concatenate and regularize with dropout? This is the cheapest option, and it is the concatenation baseline above. The file's evidence against it is indirect. Rei found that a structural auxiliary objective gave a 3.9-point $F_{0.5}$ gain beyond what a standard dropout-regularized BiLSTM-CRF achieved. MPCFN still needed loss flooding despite heavy fusion engineering. Neither result is a direct test on image-text tagging. The subsampling and shuffled-image experiments are the direct test.

Why not just use a weighted sum of views before the emissions? The scMFF result makes this a real threat, especially on tmVar. Its evidence comes from independent cell classification, though, with no structured output. A weighted sum also still routes the view through every token, so it cannot fall back to the text-only grammar when the image is irrelevant. If the weighted sum matches us on clean data and under shuffled images, the placement claim is wrong, and we will report it that way.

Why not just rely on encoder-side cross-attention, as MPCFN does? MPCFN's ablations show its gate and cross-attention help, so encoder-side fusion clearly captures some image information. The stacked experiment settles whether transition conditioning carries information the encoder misses. If adding our correction to MPCFN gives no gain, the honest conclusion is that our method is a cheap substitute for encoder-side fusion, not a complement to it.

Why not drop the CRF for independent softmax? The file shows CRF layers consistently help in these domains. Kozareva et al. report 90.92 F1 for an LSTM-CRF against 88.13 for a structured perceptron. Jagannatha and Yu report fragmented boundaries without transitions. Our mechanism also needs a transition table to modulate, so we report softmax only as a reference.

Decisive Checks

The idea stands or falls on the placement ablation. With matched parameters, the transition-conditioned model must beat the emission-conditioned one on Twitter-2015 and Twitter-2017, with the gap growing at low data. Under shuffled images it must stay within about one F1 point of the text-only CRF, while concatenation and weighted-sum fusion fall below it. If both hold, there is a cheap, inspectable fusion point that the field has ignored. If the emission placement ties, the contribution shrinks to a low-rank conditioned-transition regularizer, and we will present it as such.