Abstract
We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formulation offers two key conceptual advantages over prior neural-symbolic models for VQA. Firstly, the programs generated by our model are more understandable while requiring lesser number of teaching examples. Secondly, we show that one can pose counterfactual scenarios to the model, to probe its beliefs on the programs that could lead to a specified answer given an image. Our results on the CLEVR and SHAPES datasets verify our hypotheses, showing that the model gets better program (and answer) prediction accuracy even in the low data regime, and allows one to probe the coherence and consistency of reasoning performed.
Introduction
Visual Question Answering (VQA) is a complex task requiring a model to integrate visual perception with multi-step logical reasoning. While deep learning has achieved significant success in this domain, standard "black-box" architectures often struggle with compositional generalization—the ability to understand novel combinations of known concepts (e.g., recognizing "a red cube left of a blue sphere" after only seeing them in different relative positions).
Neural-symbolic models, such as Neural Module Networks (NMNs), address this by decomposing questions into symbolic functional programs (e.g., filter[red] -> relate[left] -> count). However, traditional NMNs are typically deterministic and require extensive "gold standard" program annotations to learn effectively. This dependency limits their scalability and makes it difficult to probe whether the model's internal reasoning is truly coherent with its output.
Probabilistic Neural-symbolic Models (Prob-NMN)
To overcome these limitations, we introduce Probabilistic Neural-symbolic Models (Prob-NMN). This framework treats reasoning programs as latent, stochastic variables. Instead of a direct mapping from question to program, Prob-NMN models the joint probability $p(x, z, a | i)$, where $x$ is the question, $z$ is the latent program, $a$ is the answer, and $i$ is the image.
The model utilizes variational inference to approximate the posterior $p(z | x, a, i)$ via an inference network $q_\phi(z|x)$. This probabilistic formulation allows the model to leverage both supervised data (with program labels) and unsupervised data (without labels) by optimizing a variational lower bound (ELBO). Training is conducted in three distinct stages:
- Question Coding: The model learns to map questions to the latent program space and back, ensuring the latent representation captures the semantic essence of the query.
- Module Training: Individual neural modules, representing functions like
filterorrelate, are trained to execute on image features. - Joint Training: The entire system is fine-tuned end-to-end. Because programs are discrete, the model uses REINFORCE to propagate gradients through the stochastic program selection process.
Experimental Evaluation
Prob-NMN was evaluated on two primary benchmarks designed to test reasoning and compositionality:
- SHAPES: A highly compositional dataset used to test performance under extreme low-supervision (using only 5% to 20% of available program labels).
- CLEVR: A standard diagnostic VQA dataset. Prob-NMN was tested using only 1,000 program-supervised examples—a tiny fraction (0.143%) of the total dataset.
- CLEVR-Humans: An out-of-distribution test set featuring natural language questions written by humans, used to assess the model's robustness to linguistic variation.
Key Results and Interpretability
Prob-NMN demonstrates significant improvements in low-data regimes. On the SHAPES dataset with 10% supervision, it achieved 94.5% accuracy compared to 62.0% for the baseline NMN. On CLEVR with only 1,000 labels, it reached 97.7% accuracy, far surpassing the baseline's 75.7%.
Beyond raw accuracy, the model offers enhanced interpretability. By forcing the model to reconstruct questions from latent programs, it learns a tighter coupling between language and logic, resulting in more "legible" (human-understandable) programs even with minimal supervision. Furthermore, the probabilistic nature enables counterfactual probing: by specifying an image and a desired answer (e.g., "No") and sampling programs $z \sim p(z | i, a)$, researchers can verify if the model's internal reasoning is coherent and sensitive to the output, rather than relying on superficial correlations.
Conclusion
Prob-NMN provides a robust, data-efficient, and interpretable approach to VQA. While it faces challenges such as optimization complexity and the need to explicitly model question premises (to avoid asking about non-existent objects), it represents a significant step toward more transparent and compositionally capable AI systems that can reason effectively with limited human guidance.
Summary updated