Cross-modal attention lets tokens from one data stream (video frames, for example) look up relevant tokens in another (audio, events, protein residues). It is the part of multimodal models that actually combines the streams, and it is expensive. The usual way to shrink it is to replace each large projection matrix with a product of two thin matrices, a low-rank factorization. Existing compression methods fit those thin matrices to the weights, or at best to the attention scores before the softmax (the function that turns scores into attention weights that sum to one). Both targets ignore two facts. The softmax discards some errors entirely and magnifies others. And in cross-modal attention the query and key tokens come from different distributions, often offset from each other by a large mean shift. We propose to compress cross-modal query-key maps by fitting the attention distributions themselves. The method starts from a closed-form two-sided solution that treats the query and key modalities separately, removes the key-mean direction that the softmax cannot see, and then refines the factors against the post-softmax output.
The practical stake is that low-rank cross-modal methods such as LMAM already report losing contextual information when they factorize linearly. The best analytical compressor for attention, A3, was built only for self-attention in language models. If fitting through the softmax recovers most of the lost accuracy at a fixed rank, efficient fusion blocks could run at much lower rank with no runtime cost. Once the factors are fitted, the compressed layer is an ordinary thin matrix product.
Core Claim
Take query-modality tokens $X \in \mathbb{R}^{N_1 \times d_1}$, key-modality tokens $Y \in \mathbb{R}^{N_2 \times d_2}$, and the bilinear score matrix $W = W_q W_k^\top$. The attention of query $i$ over keys is $p_i = \mathrm{softmax}(x_i W Y^\top/\sqrt{d})$. Adding the same constant to every score of a query leaves $p_i$ unchanged. So any part of $W$ that acts only on the key-modality mean $\bar y$ is invisible at the output. A second-order expansion of the divergence between original and compressed attention shows that logit errors are weighted by the attention covariance $\mathrm{diag}(p_i) - p_i p_i^\top$. That weighting ignores constant shifts and penalizes mistakes among high-probability keys. Our claim is that the right rank-$r$ approximation $M = UV^\top$ of $W$ minimizes
$$\mathcal{L}(U,V) = \sum_i \mathrm{KL}\!\left(p_i \,\big\|\, \mathrm{softmax}(\hat x_i U V^\top \hat Y^\top/\sqrt{d})\right) + \lambda \,\|P V_{\text{val}} - \hat P V_{\text{val}}\|_F^2,$$
where $\hat x_i, \hat Y$ are inputs produced by the already-compressed upstream layers, $P$ and $\hat P$ are the original and compressed attention maps, $V_{\text{val}}$ are the value vectors, and $\lambda$ trades distribution fidelity against output fidelity. Taking inputs from compressed upstream layers while keeping targets from the original network is the asymmetric reconstruction of Efficient and accurate approximations of nonlinear convolutional networks, which fitted CNN factors through ReLU rather than to weights. We carry that idea through a softmax.
The minimization is initialized in closed form. Under a Kronecker approximation of the paired second moments, the logit-error objective $\mathbb{E}\|X(W-M)Y^\top\|_F^2$ is solved by whitening with the query covariance $C_X$ and the key covariance $C_Y$ and taking a truncated singular value decomposition, $M^\star = C_X^{-1/2}\,\mathrm{SVD}_r(C_X^{1/2} W C_Y^{1/2})\,C_Y^{-1/2}$. We use the centered key covariance, which is where softmax invariance enters, so that no rank is spent on $\bar y$. The testable prediction is specific. The gain from centering should grow with the size of the gap between modality means, and it should be near zero for self-attention. That would explain why A3 never needed it.
Research Program
The work has three parts. The first is the estimator: centered two-sided whitening followed by a short gradient refinement of $U,V$ on $\mathcal{L}$ over a calibration set, with all other weights frozen. The refinement adds no parameters, and the result is a plain factorized matrix with zero inference overhead. This is the "mergeable" property that AuroRA secures for its non-linear fine-tuning adapters, obtained here for compression.
The second part is a diagnostic study of what compression destroys in bidirectional fusion. Gated Cross Attention for drug-target interaction showed that ungated cross-attention lets one modality wash out the other, which it tracked through cosine similarity between original and updated features. We measure that similarity, and the attention entropy, separately for each direction of a bidirectional block at each rank. The question is whether weight-space compression hurts the low-energy direction more, for example events refining frames in CDFI. If it does, the estimator should allocate rank per direction by divergence sensitivity rather than evenly.
The third part tests the estimator on three models whose attention structure differs sharply. GCA uses Sparsemax attention, which has exact zeros and ground-truth binding sites. There we replace the softmax divergence with the analogous loss on Sparsemax outputs, and whether the zero pattern survives is a direct check. HMT uses visual queries over audio keys and values. CDFI uses bidirectional frame and event attention.
Prior work plays three roles here. A3 supplies the method we extend: it computes a closed-form decomposition of the pre-softmax product $W_qW_k^\top$ for language-model self-attention. It is one-sided in spirit, since $Q$ and $K$ share a distribution, and it stops before the softmax. LMAM and Low-Rank HOCA motivate the work. Both factorize cross-modal interactions linearly and trained end to end, and LMAM names the loss of contextual information as its main limitation. CoLA constrains the design. Its input-dependent hypernetwork cannot be merged and costs throughput, so we keep every compressed operator static. We do not claim to reduce the quadratic token-pair cost of attention. The savings are in projection compute, head dimension, and cache size, and they combine freely with token pruning.
Baselines
The naive reference is the uncompressed model, together with truncated singular value decomposition applied separately to $W_q$ and $W_k$ at matched rank. The uncompressed numbers are those reported in the file. HMT reaches F-measure 0.441 on SumMe and 0.601 on TVsum HMT. GCA brings EmbDTA's Davis mean squared error from 0.438 to 0.393 and reaches nearly 90% binding-site accuracy on one head GCA. For CDFI the file gives only relative gains over frame-only trackers: +10.4% success rate and +11.9% precision rate on FE108 CDFI. We will therefore rerun the released model to obtain its absolute baseline.
The strongest existing compressor is A3's query-key solution A3, transplanted to cross-attention with the query and key covariances plugged in uncentered. It is the strongest competitor because it already optimizes a functional target in closed form and beats weight-space SVD by a large margin on LLaMA 3.1-70B: WikiText-2 perplexity 4.69 against 7.87 for SVD-LLM. No result is reported on Lacuna for A3 or any post-hoc compressor on cross-modal attention. The trained competitor is an LMAM-style linear low-rank attention trained from scratch at the same rank LMAM.
Fairness rules apply to every comparison. All methods use the same calibration set, 128 to 512 training examples drawn identically, and the same rank per layer. Any method with a gradient stage gets the same number of refinement steps and the same learning-rate search. Two ablations isolate our ingredients. The first is centered whitening with no refinement, which isolates softmax invariance. The second is refinement on the pre-softmax logit error instead of the divergence, which isolates optimizing through the nonlinearity. Beating A3 means matching its task metric at half its rank, or exceeding it by more than seed variance, taken over three calibration draws, at ranks where it degrades.
Why not just…?
Why not just fine-tune the compressed model afterwards? A short recovery fine-tune may erase the differences between initializations, and this is the most serious threat. We give every method, including plain SVD, the same recovery budget with LoRA on the remaining weights. We report accuracy against recovery steps. If the curves converge within the budget, our contribution shrinks to cheaper recovery, and we will say so.
Why not just raise the rank, or train low-rank from scratch? Raising rank gives up the compression, and the evidence from unimodal adapters is that linear factors hit a spectral ceiling. LoRAN matches full fine-tuning on MRPC (92.16% against 90.44% for LoRA) by breaking that collapse. Training from scratch, the LMAM route, requires full retraining and still loses context by LMAM's own account. The rank-sweep curves show directly where each method saturates.
Why not put a non-linear factorization like AuroRA or NLA inside the attention projection? Those methods add expressiveness to the factor, but they target adaptation or require retraining. NLA also adds inference cost when its kernels cannot be merged. Our non-linearity is the one already present, the softmax, so the deployed operator stays linear. We include AuroRA-style factors fitted with the same calibration budget as an extra arm.
Why not reduce the number of tokens instead? Token pruning attacks a different cost term and can be combined with our method. We report both separately and together on HMT so that the two savings are not conflated.
Proposed Experiments
On Davis we sweep the query-key rank for GCA's cross-attention from full down to 4. We report mean squared error, binding-site accuracy of the interpretable head, overlap of the Sparsemax support with the uncompressed model, and feature cosine similarity. On SumMe and TVsum we compress HMT's visual-to-audio blocks and report F-measure and wall-clock time. On FE108 we compress both directions of CDFI's attention blocks, report success and precision rates overall and on low-light and HDR subsets, and add the per-direction drift diagnostics.
Three checks decide the question. The first: centered whitening should beat uncentered A3 by an amount that increases with the measured gap between modality means across the three models. On a self-attention control layer the gain should vanish, and if it does not, the invariance explanation is wrong. The second: divergence refinement should outperform logit refinement mainly at low rank, where the attention is peaked, and should preserve GCA's binding-site accuracy where logit fitting does not. The third: weight-space compression should degrade the low-energy direction of CDFI's bidirectional blocks disproportionately, most visibly on low-light sequences, and divergence-based rank allocation should close that gap. If recovery fine-tuning erases all three effects at equal budget, we will report that result.
References
- Efficient and accurate approximations of nonlinear convolutional networks
- AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Object Tracking by Jointly Exploiting Frame and Event Domain
- Hierarchical multimodal transformer to summarize videos
- A3: an Analytical Low-Rank Approximation Framework for Attention
- A Low-rank Matching Attention based Cross-modal Feature Fusion Method for Conversational Emotion Recognition
- Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning
- CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks
- Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations
- Adaptive Nonlinear Compression for Large Foundation Models