Over-the-Air Consensus as Side Information for Multimodal Edge Inference

Picture several sensors in a room, such as a camera, a microphone and a wristband accelerometer, each sending its own view to a server that has to recognize an emotion or an activity. Most of what the sensors observe overlaps, so sending every feature vector separately repeats the shared part once per device. We propose to let the wireless channel carry the shared part. Each device's features are split into a part common to all modalities and a part unique to its sensor. The common parts are summed directly "in the air" by the radio channel, in a single transmission whose cost does not grow with the number of devices. Each device then sends only enough bits to describe how its full feature differs from that consensus.

The tool that makes both steps work is the nested lattice code. A lattice is a regular grid of points that is closed under addition: adding two grid points gives another grid point. A nested lattice code pairs a fine grid for precision with a coarse grid for wrap-around, so that a value can be sent as its position "modulo" the coarse grid. Closure under addition lets the receiver decode the sum of all devices' common features from their superposed radio signals. The modulo trick lets each device send its full feature at a rate set only by its distance from that sum. The idea in two sentences: disentanglement training makes the shared features agree across modalities, so their over-the-air sum is a good guess of every device's feature. Each device then pays bits only for the part the guess misses, which is the modality-specific part.

Core Claim

The claim is that a shared-versus-private split, of the kind MISA learns, can be read as a rate allocation. The shared subspace is what the channel computes for free, and the energy left in the private subspace is what each device must pay for in bits. The two lattice mechanisms enter in different roles:

What is new is chaining the two. In the second stage the side information is the decoded output of the first stage, and the encoder is trained so that this side information is close to every device's feature.

Let $x_m$ be device $m$'s feature, written as $x_m = s_m + p_m$ with shared part $s_m$ and private part $p_m$. Let $\bar s = \frac{1}{M}\sum_m s_m$ be the consensus decoded from the air over $M$ devices. Let $D$ be the allowed distortion per dimension. The standard Gaussian result for coding with decoder side information gives a per-dimension rate of roughly

$$R_m \approx \tfrac{1}{2}\log_2\!\Big(\tfrac{\mathbb{E}\|x_m - \bar s\|^2 / d}{D}\Big), \qquad x_m - \bar s = (s_m - \bar s) + p_m,$$

where $d$ is the feature dimension. Under equal channel conditions, a device sending without side information pays about $\tfrac12\log_2(\mathbb{E}\|x_m\|^2/(dD))$.

Two terms control the gap between these rates:

  • Alignment drives $s_m - \bar s$ toward zero. MISA's moment-matching loss already does this.
  • A penalty on private energy shrinks $p_m$.

The training objective is therefore

$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda\,\mathcal{L}_{\text{align}}(\{s_m\}) + \beta\sum_m \mathbb{E}\|p_m\|^2 + \gamma\,\mathcal{L}_{\text{recon}},$$

where $\lambda$, $\beta$ and $\gamma$ weight the alignment, private-energy and reconstruction terms. Sweeping $\beta$ traces an accuracy-versus-bits curve directly.

Relation to Prior Work

Three lines of work supply the method.

  • Lattice closure. The Case for Structured Random Codes in Network Capacity Theorems and compute-and-forward show that sums of lattice codewords can be decoded without separating users. That work targets message delivery, not learned features.
  • Digital over-the-air sums. Joint Design of Coding and Modulation for Digital Over-the-Air Computation builds a practical digital scheme in which superposed symbols decode to modulo sums of source values at linear complexity. We reuse it as the physical layer for the consensus stage. That paper sums generic numbers, not representations shaped to be summable.
  • Nested quantization with side information. NDQSG uses nested quantization with side information to compress correlated gradients across workers. Its side information is a running average of gradients sent over ordinary digital links, on single-modality MNIST and CIFAR-10, with no notion of a shared subspace. Our side information is produced over the air, and the encoder is trained so that the side information is good.

Disentanglement work motivates and constrains the design.

The research file found no prior paper that uses lattice codes to carry multimodal shared and private features.

Research Program

We would build four components.

  1. Encoder. Each device runs a modality encoder with shared and private projection heads, trained with the objective above. The integer-valued lattice operations are simulated during training with dither, which acts as additive uniform noise, plus a straight-through estimator.
  2. Consensus stage. Devices quantize $s_m$ onto a fine lattice with subtractive dither and transmit with lattice modulation. The server decodes the modulo sum, which lies on the lattice by closure, and divides by $M$. We start with all-ones integer coefficients and equal power. With fading, a scaling factor $\alpha$ and the self-noise term $(\alpha h - a)$ from compute-and-forward enter, where $h$ is the vector of channel gains and $a$ the integer coefficient vector. We measure how this noise propagates into $\bar s$.
  3. Residual stage. Each device sends its fine-lattice index of $x_m$ reduced modulo a coarse lattice. The coarse cell is sized from the trained residual statistics. The server recovers $x_m$ as $\bar s$ plus the modulo-reduced difference, and decoding fails whenever $\|x_m-\bar s\|$ exceeds the coarse cell. That failure rate is a measured quantity.
  4. Fusion and analysis. The server fuses the recovered $x_m$ for the task. The analysis checks whether the predicted rate in the first equation matches measured entropy-coded rates, and how rate scales with $M$.

The lattice choices are deliberately simple and practical: scaled integer grids, or products of small dense lattices, over an additive white Gaussian noise channel and simulated fading channels, as in the over-the-air computation paper.

Baselines

Naive reference. Each device quantizes and entropy-codes its full feature with no side information, and sends it over separate orthogonal channels. This is the FedMultimodal transport assumption, with concatenation and with attention fusion. It is also run with no communication limit, which gives the accuracy ceiling. All systems use the same encoder backbones, the same total channel uses per inference sample, the same SNR grid and an equal hyperparameter search budget.

Strongest accuracy competitor. At the ceiling, the strongest reported systems in the file are:

The research file has no reported result for any multimodal method under a communication budget, so the rate-constrained comparisons are new measurements. We would rerun MoNet-style encoders inside the naive orthogonal pipeline at matched bits. Beating it means matching its uncompressed accuracy within one point at a clearly lower number of channel uses.

Strongest communication competitors.

Ablations. Two ablations isolate the key ingredient:

  • Setting $\lambda=0$ removes alignment, so the side information is poor.
  • Setting $\beta=0$ removes pressure on private energy.

Beating these means a lower measured rate at equal accuracy across SNRs, with the gap predicted by the residual-variance term.

Why not just…?

Why not just use nested quantization with side information on the raw features, without disentanglement? This is the most serious threat. Wyner–Ziv coding, the information-theoretic recipe for compressing a signal when the decoder holds correlated side information, already exploits correlation without any explicit split. The answer depends on how well the averaged raw features predict each device's features. CREMA-D audio and video encoders trained separately live in unrelated coordinates, so their average should predict either one poorly. That would make the residual large and the coarse-cell decoding failures frequent. The $\lambda=0$ ablation tests this directly. If the ablation reaches the same rate, our claim reduces to "nested quantization over the air works," and the disentanglement component should be dropped.

Why not just sum the shared features with analog over-the-air computation? At high SNR with accurate phase compensation, analog summation may match lattice summation at lower complexity. The over-the-air computation paper reports that uncoded schemes degrade under phase shifts and noise, but it gives no numbers on learned features. We compare both across SNR and phase-error levels. If analog wins at high SNR, we report the crossover rather than claim a uniform gain.

Why not just send only the fused prediction or a small fused vector from one device? Fusion requires all modalities in one place. Sending raw features to a single fusing device costs the same orthogonal transmissions as the naive baseline. Fusing late at the server is the setting where per-device bits matter.

Why not just let MoNet-style decoupled training fix imbalance and skip structured coding? MoNet addresses accuracy, not bits. It is therefore our accuracy target rather than a competitor on rate, and we test whether adding the private-energy penalty undoes its gains on the weak modality.

Proposed Experiments

Datasets.

  • CREMA-D with audio and video as two devices.
  • KU-HAR with accelerometer and gyroscope, using the FedMultimodal partitions and its 30% missing-modality condition.
  • CMU-MOSI and CMU-MOSEI with three devices.

Metrics.

  • Task accuracy or F1 against total channel uses per sample, at SNRs from low to high, on additive white Gaussian noise and simulated fading channels.
  • Measured entropy-coded rate per device, compared with the residual-variance prediction.
  • Coarse-cell decoding failure rate.
  • Per-modality encoder accuracy, to detect suppression of weak modalities.
  • Rate as $M$ grows, by adding synthetic redundant views.

Decisive checks.

  1. Measured rate should track $\tfrac12\log_2$ of the residual variance across the $\beta$ sweep. If it does not, the mechanism is not what drives the savings.
  2. The disentangled system must beat the $\lambda=0$ ablation at matched accuracy, or the coupling to disentanglement is unnecessary.
  3. Consensus cost must stay flat as $M$ increases while orthogonal cost grows linearly.
  4. Under missing modalities, the consensus is averaged over the devices present. Accuracy loss should be no worse than the FedMultimodal reference.