Multimodal models that read images, audio, or video alongside text tend to lean on one input type and quietly ignore the others. This failure is called modality dominance. A compression study, MBQ, measured it from the opposite direction: in vision-language models, the training loss reacts more than ten times as strongly to language tokens as to vision tokens. Vision tokens can therefore be quantized harder (stored and computed with fewer bits) at almost no accuracy cost. Chip designers could exploit this directly. A design-space search could build an accelerator that runs language tokens at high precision and vision or audio tokens at low precision, gaining power and area. There is a catch. The low sensitivity of vision tokens may be partly a symptom of dominance itself: the model barely uses the visual evidence. A chip tuned to that sensitivity could keep average benchmark accuracy intact while further destroying the model's ability to use the weaker modality in exactly the cases where it matters, namely when text and image disagree.
This note proposes a design-space search for multimodal accelerators that adds a fourth objective to power, performance, and area. Alongside ordinary accuracy, it measures accuracy on counterfactual, cross-modal conflict questions, where the model must trust the image or sound over the text. The design space is one where precision and token budget are assigned per modality. The bet is twofold. First, modality-asymmetric hardware is a large efficiency win. Second, the designs that look best on aggregate accuracy are systematically worse on the dominance-sensitive axis. Choosing among them therefore requires measuring that axis explicitly.
Core Claim
Hardware design-space exploration (automated search over chip configurations that traces the set of designs where no objective can improve without another getting worse, the Pareto front) has never been aimed at multimodal workloads. Compilers such as SEGA-DCIM generate compute-in-memory macros (arrays that compute matrix products inside the memory itself) from parameterized templates. SEGA-DCIM searches over array shapes and precisions with a genetic algorithm, but it treats the network as a uniform stream of matrix multiplications at one precision. Modality-dominance work, by contrast, stays entirely in software. MBQ writes a GPU kernel, and MACER prunes audio and video tokens with separate budgets.
We make two claims. The first is that a bit-serial compute-in-memory datapath can vary activation precision token by token, using a modality tag, at near-zero area cost. Bit-serial means the array processes one bit of the input per cycle, so a 4-bit token simply takes half as many cycles as an 8-bit one. This exposes per-layer, per-modality precision and per-modality token retention as hardware knobs. The second claim is that once these knobs exist, optimizing them against aggregate accuracy selects designs that degrade counterfactual cross-modal accuracy. The proposal predicts that on the C$^3$PO counterfactual tiers, designs on the three-objective front plus aggregate accuracy will be dominated by designs found when counterfactual accuracy is a separate objective.
The second claim is falsifiable and is the point of the work. If it fails, the result is still a useful but more ordinary contribution: modality-asymmetric precision search.
Research Program
Hardware template. We extend a SEGA-DCIM-style digital compute-in-memory template. Activations are fed bit-serially, and each token carries a modality tag that sets how many bit-cycles it receives. Weights stay shared and uniform per layer, because in early-fusion decoders the same weights process every token type. The asymmetry therefore lives on the activation side, plus a per-modality token buffer whose size is set by a retention ratio in the style of MACER.
Search space. The searched variables are:
- array rows and columns;
- input and output buffer sizes;
- weight precision per layer;
- activation precision per layer and per modality (language, vision, audio);
- token retention ratio per modality.
Hardware cost model. Energy, latency, and area come from an analytical model of the template. It is calibrated against a few dozen configurations pushed through logic synthesis and place-and-route on a 28nm-class process, the same flow class SEGA-DCIM reports.
Accuracy model. Accuracy comes from simulated quantization of the real model under each precision and retention assignment. Calibration uses MBQ's modality-weighted reconstruction loss, so that the software side starts from the strongest existing recipe.
Objectives. The search optimizes five objectives with NSGA-II, as SEGA-DCIM does:
- energy per query;
- latency;
- area;
- aggregate accuracy;
- dominance-sensitive accuracy, defined as accuracy on C$^3$PO's counterfactual-conflict subset together with a reliance score (how often the answer changes when the non-text input is swapped for a mismatched one).
Cheap screening. Because each accuracy evaluation is a full benchmark pass, a small surrogate model screens candidates, and only promising ones get a full evaluation.
Models. We use LLaVA-onevision-7B, which MBQ studied, for vision-language. We use Qwen2.5-Omni-7B, which MACER studied, for audio-video.
Baselines
The naive reference point is a uniform-precision accelerator from the same template, searched only over power, performance, and area. It runs INT8 activations for every token with MBQ-calibrated 4-bit weights. This is what SEGA-DCIM-style compilers would produce today. SEGA-DCIM reports 22 TOPS/W and 1.9 TOPS/mm² for its INT8 macro at 28nm. We use those numbers only as a sanity anchor for our cost model, not as a competitor on accuracy.
The strongest existing software method on the accuracy side is MBQ itself. MBQ reports gains of up to 4.4% under W3A16 and 11.6% under W4A8 over AWQ and SmoothQuant. MBQ applies one precision to all tokens and weights the loss by modality. We run it with the same 128-pair calibration set it uses and place it on our template at uniform precision. Beating it means matching its aggregate accuracy at lower energy, and doing so without losing counterfactual accuracy.
The strongest token-side competitor is MACER. It reports 98.7% of full-token accuracy at 25% retention on Qwen2.5-Omni-7B. We implement its fixed audio and video budgets on the uniform-precision template at iso-energy with our designs.
A per-layer mixed-precision search without modality tags, in the spirit of QADAM, gets the same search budget, measured in evaluated configurations. It isolates whether the modality tag matters at all.
Two ablations isolate the key ingredients. The first runs our full search with the dominance objective removed. The second runs it with modality tags but fixed retention.
No reported C$^3$PO result was found on Lacuna for LLaVA-onevision or Qwen2.5-Omni, so we measure the full-precision reference ourselves. For context, C$^3$PO reports 43.98% for the strongest open model, Qwen3-Omni, and near-random scores (about 25%) on higher counterfactual tiers. That second number warns that floor effects could hide degradation. We will therefore report the lower tiers separately.
Why not just…?
Why not just prune the weak modality harder and keep uniform hardware? If vision tokens are redundant, removing them may beat quantizing them, and MACER already does this well. This is a real threat. We compare, at matched energy, (a) pruning only, (b) asymmetric precision only, and (c) both. If (a) dominates, the precision knob is not worth building. Our expectation is that pruning removes visual coverage that counterfactual questions need, while lower precision keeps it. That expectation is a hypothesis, not established.
Why not just use per-layer mixed precision, which existing accelerator search already supports? Per-layer search cannot tell a language token from a vision token, because both pass through the same layer. The non-tagged search baseline above answers this directly. If it reaches the same front, the modality tag is unnecessary.
Why not fix dominance in training first, with CAT-GS or MMPareto, and then compress normally? CAT-GS and MMPareto do rebalance learning: MMPareto reaches 75.13% on CREMA-D versus 71.10% for uniform averaging. However, both retrain the model with unimodal losses, which is costly for 7B omni models. More importantly, retraining does not remove the question of whether compression later undoes the balance. As a small-scale check, we will train a CREMA-D audio-visual model with and without MMPareto, run our search on both, and see whether the dominance objective still changes the selected hardware.
Why not just evaluate on a GPU like MBQ? A GPU cannot run per-token variable activation bit-widths cheaply. The point of the proposal is that bit-serial compute-in-memory can, so the question cannot be answered there.
Proposed Experiments
Experiment 1 (the decisive check). We take the uniform and the asymmetric designs that are Pareto-optimal on energy, latency, area, and aggregate accuracy. We measure them on C$^3$PO counterfactual tiers and on the swap-based reliance score. Aggregate accuracy is measured on MMMU, OCRBench, TextVQA, and SEED-Bench for LLaVA-onevision, and on DailyOmni, WorldSense, and Video-MME for Qwen2.5-Omni.
Experiment 1 succeeds if asymmetric designs lose meaningfully more counterfactual accuracy than uniform designs at equal aggregate accuracy. "Meaningfully" means more than the seed-to-seed spread across calibration resamples. If they do not, the dominance objective is dropped and the study reports the efficiency result alone.
Experiment 2. We run the full five-objective search and compare its front against the four-objective front. The comparison uses hypervolume, the volume of objective space dominated by a front. We also report how much energy must be spent to recover counterfactual accuracy, for example whether keeping vision activations at 8 bits in a few middle layers suffices.
Experiment 3. We check the cost model against synthesized designs. We report the area overhead of modality-tagged bit-serial control, and the error between predicted and post-layout energy on the held-out configurations.
Experiment 4. We repeat Experiment 1 on Qwen2.5-Omni, where the dominant input is audio rather than text. MACER reports that audio draws excess attention there, which tests whether the effect is about whichever modality dominates rather than about text specifically.
References
- MBQ: Modality-Balanced Quantization for Large Vision-Language Models
- SEGA-DCIM: Design Space Exploration-Guided Automatic Digital CIM Compiler with Multiple Precision Support
- Allocation Before Ranking: Decoupled Token Compression for OmniLLMs
- C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
- QADAM: Quantization-Aware DNN Accelerator Modeling for Pareto-Optimality
- CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance