Answering a physical "what if" question, such as "what would happen if this container were dropped?" or "which of these two objects would dent the other?", depends on properties a camera cannot see: mass, hollowness, material. Audio often supplies them. A tap on wood sounds different from a tap on plastic, even when the two objects look the same. When the microphone fails, current systems try to reconstruct the missing audio features from the video. For properties that are visually invisible, that reconstruction cannot be done. The system ends up producing a visual stereotype of the sound and then answers with full confidence. We propose a different approach. The system keeps an explicit probability distribution over hidden physical properties, conditioned on whichever sensors are actually present. It then answers counterfactual questions by averaging physical outcomes over that distribution. When the missing sensor carried the decisive information, the answer correctly becomes uncertain instead of confidently wrong.
Training such a distribution requires ground truth about mass and material, and real video does not provide it. A physics simulator does. Synthetic data is therefore the natural training ground. We also argue that a low-dimensional physical interface ("probably hollow plastic, mass 40 to 80 grams") should transfer from simulation to reality better than the pixel- and spectrogram-level features that current reconstruction modules pass along. In this proposal counterfactual reasoning is the core mechanism, missing modalities are the stress regime, and the simulation-to-reality gap is the constraint that decides whether the design is useful.
Core Claim
A missing modality in physical reasoning is an identifiability problem, not a feature-filling problem. Some hidden properties can be recovered from any sensor subset. Others become unrecoverable once a particular sensor is gone. A model that represents this explicitly should behave better than one that fills in missing features. Write $x_S$ for the observed signals from sensor subset $S$, $z$ for the hidden physical properties of the objects, $I$ for a hypothetical intervention, and $a$ for the answer. The model then computes
$$ p(a \mid x_S, \mathrm{do}(I)) = \int p(a \mid z, \mathrm{do}(I)) \, p(z \mid x_S) \, dz , $$
where $p(z \mid x_S)$ is a posterior over properties trained in simulation for every sensor subset, and $p(a \mid z, \mathrm{do}(I))$ comes from rolling a learned dynamics model forward under the intervention.
The claim is testable in three ways. On real audiovisual data with audio removed, the model should keep its accuracy on visually decidable questions, become calibrated rather than wrong on visually ambiguous ones, and beat feature-filling methods on selective accuracy. Selective accuracy here means accuracy on the questions the model chooses to answer confidently.
Relation to Prior Work
The closest existing system is Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning (RDCL). It splits video and audio into time-invariant "static" factors (material, shape) and time-varying "dynamic" factors (actions). It applies a causal "total indirect effect" to ask how answers change under altered properties. It adds a plug-in module that recovers a missing modality by projecting the available one into a shared space. RDCL motivates this proposal: it shows missing modalities matter on physical questions. It also differs from our design on the two points that matter most. First, its recovery module outputs a single reconstructed feature, not a distribution over properties. Second, it is trained and tested only on real recordings, never in simulation. Its predecessor, Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning, established audio as the tie-breaker when appearance is ambiguous but has no recovery mechanism at all. PACS: A Dataset for Physical Audiovisual CommonSense Reasoning supplies the real evaluation data. It also supplies the clearest example of why reconstructing audio from vision is ill-posed: real versus hollow plastic fruit.
A second line of work supplies the method. It infers hidden properties from observed interactions and then simulates counterfactual futures. CoPhy: Counterfactual Learning of Physical Dynamics estimates latent mass and friction from a context video with graph networks and rolls out the counterfactual trajectory. ComPhy: Compositional Physical Reasoning of Objects and Events from Videos learns mass and charge from reference videos and applies them to new scenes. Both use point estimates, both are vision-only, and both stay in simulation. CoPhy names real-world transfer as its main open limitation. We reuse their "infer properties, then simulate" structure. We replace the point estimate with a posterior conditioned on the available sensors.
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning is the only work in our file that carries physical concepts from synthetic training to real video (the Block Tower Dataset). It only classifies "falling" and runs no counterfactual simulation on real footage.
Two results constrain the design. CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering finds that shifting physical constants collapses models toward chance. It also finds that accuracy drops from the first collision to later ones in a chain. We therefore keep rollouts short and evaluate under shifted property ranges. Second, the simulation literature notes that synthesizing accurate impact acoustics is expensive and approximate. This is the reason we pass only coarse physical posteriors from audio into reasoning and do not pass synthetic audio features.
Research Program
We would build a synthetic audiovisual counterfactual dataset in ThreeDWorld, the engine behind CRIPP-VQA. Scenes would contain tapped, dropped, rolled, and colliding objects whose material, hollowness, mass, and restitution are sampled independently of their visual appearance. Each scene comes with rendered video, impact audio from modal sound synthesis, the ground-truth properties, and counterfactual outcomes under interventions such as "swap mass," "remove an object," or "drop from twice the height." We would deliberately include visually identical objects with different interiors. This makes the identifiability structure explicit: for each question type and each sensor subset, the simulator tells us whether the answer is determined.
The model has three parts. The first is a set of per-modality encoders initialized from pretrained audio and video encoders, the same family RDCL builds on. The second is a posterior network that takes whichever encoder outputs are present and outputs a mixture distribution over per-object properties. It is trained with likelihood on simulator ground truth under every sensor subset, so it never sees zero-filled inputs. The third is a short-horizon learned dynamics model in the CoPhy style that answers questions by rolling out samples from the posterior under the intervention.
To bridge to reality, we would apply visual and acoustic randomization in simulation. We would then fine-tune the encoders and the answer head on the real PACS training split, keeping the posterior's output space fixed as physical units. A diagnostic arm would ask whether real-world failures come from perception (the posterior is wrong) or from dynamics (the rollout is wrong). It would do this by measuring posterior quality on a small set of real objects we weigh and label by hand. That is the rendering-versus-physics split which the simulation literature says few benchmarks provide.
Baselines
The naive reference point is a fusion model trained on PACS with standard modality dropout as in ModDrop: Adaptive Multi-Modal Gesture Recognition, with missing audio zero-filled at test time. It shows what you get with no recovery and no simulation. Beating it is necessary but not interesting.
The strongest existing competitor is RDCL with AudioCLIP features, which reports 65.3% accuracy on full PACS (Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning). For missing modalities, RDCL reports an 11.8% gain over disentangled models without its recovery module. We found no absolute audio-missing accuracy for RDCL on Lacuna, so we will rerun its released recipe ourselves. It is the strongest competitor because it already combines counterfactual reasoning with missing-modality recovery on exactly this data. For fairness, RDCL receives the same pretrained encoders, the same real PACS training data, and an equal hyperparameter search budget measured in GPU-hours. In a second arm it also receives the same synthetic pretraining data, routed through its own reconstruction module. That arm separates the value of simulation from the value of the posterior.
On the synthetic side, the reference numbers are 63.21% counterfactual accuracy for Aloe* with BERT on CRIPP-VQA and a mean pixel error of 28.25 for CoPhyNet on BlocktowerCF, both from those papers. We will match or approach them before claiming anything about transfer.
The decisive ablation keeps our full pipeline but collapses the posterior to its mean before the rollout. This turns the model into a point-estimate feature-filler with identical data and capacity, and it isolates whether representing uncertainty is what helps. Large vision-language models (GPT-4V at 51.3% and Qwen-VL at 55.7% on PACS, as reported by RDCL) serve as an external reference rather than the main competitor.
Beating RDCL means three things on the audio-missing condition. Accuracy must be no worse overall. Expected calibration error and Brier score must be clearly lower. Selective accuracy must be higher at matched coverage on the ambiguous subset described below. With audio present, accuracy must be within noise of RDCL.
Why not just…?
Why not just fill in the missing audio features, as RDCL already does? For questions where vision determines the property, filling in is fine, and we expect no advantage there. For visually identical objects with different interiors, any deterministic filler must output the same audio feature for both. By construction it then answers at most one of them correctly and assigns high confidence to both. The mean-collapse ablation tests exactly this. If it matches the full posterior model on calibration in the ambiguous subset, our central claim is wrong.
Why not just skip simulation and train on real PACS, which is already labeled? This is the most serious threat. PACS labels answers, not masses or materials, so it cannot directly supervise a property posterior. It is also unclear whether the gains would come from simulation or simply from more data. We will settle this with a data-matched comparison. Our model will be trained with no synthetic pretraining and its posterior supervised only through answer loss. We will also vary the real training fraction from 10% to 100%. If simulation helps only below some data fraction, or not at all, we will report that and narrow the claim to the low-data regime.
Why not just prompt a large vision-language model, which has web-scale commonsense? On PACS, GPT-4V scores 51.3%, and RDCL reports such models attending to brand logos rather than material. Perception Test shows a zero-shot model at 46.2% against humans at 91.4% (Perception Test: A Diagnostic Benchmark for Multimodal Video Models). We include one such model on the audio-missing split mainly to check whether its stated uncertainty is calibrated. We do not expect it to be.
Why not just use domain randomization with an end-to-end model and no explicit physical properties? Randomization narrows the appearance gap but leaves the model free to rely on simulator regularities. That is the failure CRIPP-VQA documents under shifted constants. We run this end-to-end variant with the same synthetic data. We compare it on PACS and on synthetic test sets whose property ranges lie outside training.
Proposed Experiments
The first experiment runs in simulation. We test whether the posterior has correct coverage for every sensor subset, meaning its 90% intervals contain the true mass about 90% of the time. We also test whether counterfactual accuracy on CRIPP-VQA-style and CoPhy-style questions holds up with audio removed for audio-identifiable properties. The same tests run under the out-of-range property splits.
The main experiment is on real PACS and PACS-Material, evaluated under three conditions: audio present, audio removed, and audio corrupted with noise. We also define an ambiguous subset, consisting of question pairs that audiovisual annotators answer correctly but video-only annotators do not. We will collect fresh video-only human judgments on the test split, following the protocol behind PACS's reported 90.5% visual-only and 96.3% audiovisual human accuracies. We report accuracy, expected calibration error, Brier score, and the area under the risk-coverage curve, both overall and on the ambiguous subset. The decisive check is the ambiguous subset with audio removed. The full model should be markedly better calibrated and more selective than both RDCL and the mean-collapse ablation, while matching them on the unambiguous remainder.
The third experiment is the diagnostic arm on hand-labeled real objects, which attributes remaining errors to perception or to dynamics. The fourth checks whether the same posterior helps the Block Tower transfer go beyond "falling" classification to short counterfactual stability judgments in real video.
A clean negative result is also informative. If the mean-collapse ablation matches the full model, or if real-only training matches simulation pretraining, the field learns that missing physical modalities can be treated as a filling problem after all.
References
- Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning
- Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning
- PACS: A Dataset for Physical Audiovisual CommonSense Reasoning
- CoPhy: Counterfactual Learning of Physical Dynamics
- ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
- Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
- CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering
- ModDrop: adaptive multi-modal gesture recognition
- Perception Test: A Diagnostic Benchmark for Multimodal Video Models