Multimodal prediction systems are usually trained as one piece. All the encoders for images, text, sensor streams and lab values are optimized together, so none can be replaced, upgraded or added later without retraining the rest. This note proposes making the hidden state that passes between modules a fixed, known geometry chosen before any training. Every module reads and writes class beliefs as points relative to a frozen set of maximally separated class directions, a simplex ETF. Because that coordinate system is known in advance, a team can build a new modality module on its own data, with no access to the other modules. It does this by feeding the module synthetic "incoming states" sampled from the known geometry. The finished module can then be composed with modules it has never met.
A simplex ETF (equiangular tight frame) is a set of $K$ equal-length unit directions, one per class, with every pair at the same, largest possible angle. The claim is concrete. Fixed geometry is the property that makes a module's upstream input simulable, and simulability is what turns independently developed modules into interchangeable parts. The test is whether such separately built modules, composed in any order and with any subset missing, approach a jointly trained system. A second test is whether they beat the obvious modular alternative, which is averaging the predictions of independent unimodal classifiers, on calibration and missing-modality robustness.
Where this sits
The ingredients exist separately. MultiMoDN is the modular architecture this note supplies. In it, each modality encoder updates a shared state vector, and task decoders read that state. A missing modality's module is simply skipped, which gave robustness to non-random missingness and a 20% AUROC gain on Weather2k. Its decoders are learned jointly with its encoders, though. The state space is therefore an artifact of one training run, and a module trained anywhere else cannot be plugged in.
Three papers motivate the move to fixed geometry, each showing that a frozen class geometry acts as a shared coordinate frame across components that never trained together:
- No Fear of Classifier Biases (FedETF) fixes one ETF across federated clients and adds a small learnable projection in front of it. This lifted global CIFAR-10 accuracy at Dirichlet skew $\alpha=0.05$ from 44.48% (FedAvg) to 56.08%.
- Stationary Representations proves that learnable classifiers cannot meet formal backward-compatibility constraints. It then uses a fixed simplex to hot-swap backbones across architecture families.
- CoReS does the same for sequential model upgrades, reaching average compatibility 0.90 versus 0.51 for BCT on CIFAR-100.
All three apply the shared frame to one modality and one feature extractor at a time. None uses it as the interface between different modules inside a single prediction. None exploits the fact that a known target geometry lets a module be trained against a simulated partner.
Neural Collapse in Multi-Task Learning supplies the multi-task layout. Neural collapse is the tendency of class features to converge onto such symmetric configurations late in training. That paper shows separate task heads converge to mutually orthogonal ETFs, with the shared feature becoming a scaled sum of each task's class vertex. It also shows freezing those heads in advance costs no accuracy. Its setting is a monolithic shared backbone, whereas we use its geometry as the format of a multi-task interface state.
Two papers supply the theoretical cover for freezing the geometry up front. On the Optimization Landscape of Neural Collapse under MSE Loss argues that the ETF is the natural endpoint of training, so freezing it early costs little. Maximum Class Separation as Inductive Bias in One Matrix shows a fixed separation matrix helps under class imbalance; on long-tailed CIFAR-10, accuracy rose from 56.98% to 69.70%. Imbalance is common in the clinical data we target.
On the systems side, toolboxes like FaceX-Zoo and the schema-based OmniEvaluator enforce syntactic contracts such as tensor shapes and method signatures. They do not enforce contracts on what the tensors mean. Our proposal is a semantic contract that can be checked mechanically.
Core Claim
Let each task $t$ have $K_t$ classes with fixed unit vertices $w^{t}_1,\dots,w^{t}_{K_t}$. Each task's vertices form a simplex ETF inside its own subspace, and the subspaces of different tasks are mutually orthogonal, following Wang et al. The state $s\in\mathbb{R}^d$ is read by a parameter-free decoder:
$$p_t(k\mid s)=\mathrm{softmax}_k\big(\tau\,\langle s, w^{t}_k\rangle\big),$$
where $\tau$ is a single shared temperature. A modality module $m$ maps its input and the incoming state to an updated state, $s' = s + g_m(x_m, s)$.
The contract states that the incoming state can be any point in a known family. For a sample whose task-$t$ label is $y_t$, we simulate incoming states as
$$\tilde s=\sum_t \Big(\alpha_t\, w^{t}_{y_t} + \beta_t \sum_k \pi^{t}_k w^{t}_k\Big)+\epsilon .$$
Here $\alpha_t\ge 0$ is how confident the upstream modules already are about the correct class. The vector $\pi^{t}$ is a random belief spread over classes, which represents upstream confusion. The scale $\beta_t\ge 0$ sets how much of that confusion is present, and $\epsilon$ is small isotropic noise. The empty state $s=0$ represents "first module in the chain."
A module trained on its own data alone sees $\tilde s$ in place of real upstream states. It is trained so that the decoder's prediction after its update is correct and calibrated. It must also not erase confident correct beliefs it cannot improve on.
The prediction is as follows. Because every other module is trained to emit states in this same family, the simulated distribution covers the states a module meets at deployment. Separately built modules should then compose in any order and under any missingness. With learned decoders this cannot happen, because the state family is unknown until joint training ends.
Research Program
Interface and acceptance test. We will build the interface as a small library: a fixed-geometry state type, the parameter-free multi-task decoder, and the simulated-state sampler. We will also build an acceptance test that any submitted module must pass before composition. The test checks three things on held-out simulated states: outputs stay inside the family, measured by norm and by alignment with the vertices; predictions are calibrated; and confident correct incoming beliefs are not degraded. This is the modular-systems contribution: a semantic interface contract checked by a test, analogous to an abstract base class.
Simulator calibration. We will study how the simulator's parameters, the ranges of $\alpha$ and $\beta$ and the noise level, must be chosen. The measurable quantity is a distribution gap: the distance between simulated states and the real states produced by actual upstream modules. We will report how composed accuracy degrades as this gap grows. This tells practitioners when separately built modules can be trusted.
Dimension constraint. The $d\ge \sum_t (K_t-1)$ requirement noted by CoReS is benign here, because clinical and sensor tasks have few classes. We state it as a scope limit rather than solve it.
Baselines
Jointly trained MultiMoDN. This is the reference for what modularity costs. It is MultiMoDN with learned decoders, trained end to end with all modalities available, using the released architecture and the same encoders. The file reports only relative results for it: AUROC "nearly identical" to parallel fusion on MIMIC-IV and EDU, and a 20% AUROC gain on Weather2k. No absolute AUROC figure is reported on Lacuna for these datasets, so we will rerun it ourselves. Success means our separately built modules come within a small, pre-registered margin of its AUROC when all modalities are present, and match or exceed it under missingness.
Late fusion. This is the strongest modular competitor. Each modality gets an independent classifier, and at test time their log-probabilities are averaged, or summed as a product of experts, over whichever modalities are present. It is equally modular and needs no shared geometry, which makes it the main threat. Each unimodal classifier gets the same encoder, data, epochs and hyperparameter search budget as our module. We will also give it temperature scaling fitted on a validation split, so the calibration comparison is fair.
Naive composition with learned decoders. This is the "do nothing" reference and should fail. Independently trained MultiMoDN modules, each with its own learned decoder, are chained at test time, with one decoder chosen arbitrarily. It shows how incompatible separately learned state spaces are.
Two ablations. The first isolates the simulator: modules use the fixed ETF but are trained only from the empty state, never on simulated incoming states. The second isolates the geometry: a random fixed orthonormal class frame replaces the ETF.
Alignment-map baseline. This represents the BCT and translation-network family. Independently trained modules are reconciled after the fact by a linear map learned on a small set of samples where both modalities are observed. We test it at matched paired-data budgets.
Why not just…?
Why not just average the predictions of independent unimodal classifiers? This is a genuine threat, and for accuracy with all modalities present it may tie. Product-of-experts fusion assumes each modality's evidence is independent given the label. When modalities are redundant, as with correlated weather sensors or overlapping clinical signals, that assumption makes the fused prediction double-count evidence and become overconfident. A module that sees the incoming belief can learn to shrink its update when the belief is already confident.
We will settle this with two tests. The first is a controlled two-view benchmark where we dial redundancy between views from zero to full. The second compares expected calibration error and AUROC on Weather2k and MIMIC-IV across every missing-modality subset. If late fusion matches us on calibration as well, we will report that the modular value of fixed geometry reduces to calibrated late fusion for classification. That result would itself bound the claims made in the compatibility literature.
Why not just train jointly with random modality dropout? Dropout during joint training gives robustness to missingness, but it still needs every modality present at training time, together, in one run. It cannot absorb a sensor added next year or an encoder rebuilt by another team without retraining. We include it as an upper reference and measure the cost of separate development against it. We do not claim to beat it under full joint access.
Why not just learn a mapping between independently trained state spaces? Mapping networks need paired, co-occurring data, and in pairwise backward-compatible training they degrade over successive updates. Stationary Representations reports BCT and CVS compatibility collapsing toward zero over 31 tasks, while the fixed simplex held at 0.65. We test the mapping baseline at matched paired-data budgets. Our method uses zero paired samples to build modules, so a win for mapping only at large paired budgets would still favor our approach in the stated setting.
Proposed Experiments
Datasets. We use the three MultiMoDN datasets: MIMIC-IV, combining images, notes and chart events; EDU, an educational interaction time series; and Weather2k, meteorological sensors. These are multi-task, which lets the orthogonal task subspaces be tested directly. We add a controlled two-view CIFAR-10 benchmark, splitting each image into complementary crops with an adjustable overlap that sets redundancy.
Protocol. Each modality module is trained by a separate process that sees only its own modality's data and the simulator. Composition happens only at evaluation, over all orders and all non-empty subsets of modalities.
Metrics. We report AUROC and accuracy, expected calibration error, variance across composition orders, and the simulated-to-real state gap.
Hot-swap test. One encoder, for example MIMIC-IV notes, is replaced by an architecturally different module built separately. We then measure the change in fused performance with no retraining of any other component.
What counts as decisive. Three results together would support the claim: the simulator ablation loses clearly to the full method under composition; the method stays within margin of jointly trained MultiMoDN; and it beats calibrated late fusion on calibration in the redundant settings. If the simulator ablation performs as well as the full method, the simulation idea is unnecessary. If late fusion ties everywhere, the fixed-geometry interface adds nothing for classification.
References
- MultiMoDN - Multimodal, Multi-Task, Interpretable Modular Networks
- No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier
- Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements
- CoReS: Compatible Representations via Stationarity
- Neural Collapse in Multi-Task Learning
- On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features
- Maximum Class Separation as Inductive Bias in One Matrix
- FaceX-Zoo: A PyTorch Toolbox for Face Recognition
- A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation