Abstract
Deep learning models are notoriously opaque, but what if we could understand them by letting them ask questions? This paper introduces "asking networks," a framework where a model iteratively requests specific hints about parts of a task it finds confusing. Applied to image colorization, the model naturally learns to ask for the colors of semantically meaningful objects, revealing its inner uncertainties and outperforming traditional interactive colorization methods.
Introduction
Deep neural networks are often treated as black boxes. While post-hoc visualization tools like saliency maps can highlight which input pixels influenced a decision, they fail to communicate what information the model feels it is missing. In multimodal "one-to-many" tasks—such as image colorization, where a single grayscale image could have many valid color palettes—it is difficult to discern why a model chose a specific output. This research addresses the interpretability gap by shifting the paradigm: instead of just observing the model passively, the authors allow the model to communicate its uncertainty by asking questions during the prediction process.
Methods: The Asking Network Framework
The authors propose a general framework called asking networks. In this setup, a model does not simply output a final prediction; instead, it sequentially generates a "question" and receives an "answer" before refining its guess.
For the task of image colorization, the model outputs a spatial heatmap (values from 0 to 1) indicating a specific region it wants to know the color of. An external function then provides the average ground-truth color of that region. Using a U-Net architecture, the model updates its intermediate state and asks subsequent questions based on the new information.
To ensure the model asks interpretable and useful questions, two training techniques are employed:
- Smoothing Loss: Encourages the model to generate spatially coherent heatmaps rather than scattered pixels.
- Noise Injection: Prevents the model from asking for the average color of multiple distinct objects by injecting random noise into the answers, forcing it to isolate single, distinct objects for reliable hints.
Key Insights and Results
The model exhibits "greedy curiosity," consistently targeting high-error regions first. Interestingly, the act of formulating useful questions leads to emergent segmentation; the model learns to identify semantically coherent objects (like a person's shirt) because they provide the most mathematically useful hints for minimizing loss.
Quantitatively, the asking network significantly outperforms existing interactive colorization baselines. With just one hint, the proposed model achieves a PSNR of 28.16, compared to 25.37 for the next best baseline. Furthermore, on the VOC 2012 dataset, the regions the model asks about overlap with a single ground-truth semantic class 86.7% of the time, proving that the questions are semantically meaningful to humans.
Conclusion
While training asking networks is computationally intensive, the framework provides a novel path toward interpretable AI. By allowing models to "ask," we gain insight into their internal decision-making and provide a mechanism for human users to steer outputs. Future work may extend this paradigm to other domains, such as image inpainting or natural language processing, where resolving ambiguity is critical.
Summary updated