TL;DR
This paper provides a novel probabilistic interpretation of the attention mechanism, showing that standard dot-product attention is actually a special case of Maximum A Posteriori (MAP) inference. By reframing attention as a generative mixture model, the authors introduce a principled way to update model parameters in real-time using Expectation-Maximization (EM), significantly improving how interactive segmentation models respond to human feedback.
The Challenge of Interactive Adaptation
Interactive semantic segmentation is a collaborative process between a human and a machine. An annotator provides "clicks" (positive or negative cues) to correct a model's mask, and the model must update its prediction accordingly.
However, standard deep learning models are often brittle in this setting. They might ignore a click's context or fail to propagate that correction to similar-looking pixels across the image. The fundamental challenge is online adaptation: making a pre-trained model learn from new, sparse information provided at inference time without a full, slow retraining process.
The authors argue that the standard attention mechanism—the engine behind Transformers—is usually treated as a deterministic black box. By giving it a probabilistic foundation, they unlock a way to update the model's memory on the fly.
Probabilistic Attention Framework
The core contribution is a probabilistic generative model for queries and values.
- Attention as Inference: The authors hypothesize that a model contains a bank of "probabilistic memory units." Generating a query/value pair involves sampling a unit and then sampling from that unit's specific distribution. They prove that assuming these distributions are Gaussian makes the standard dot-product attention update mathematically equivalent to performing MAP inference.
- Key Adaptation (Unsupervised): Using the Expectation-Maximization (EM) algorithm, the model can adapt its keys to better match the queries of a specific image at test time. This helps the model account for domain shift when the test image differs slightly from the training data.
- Value Propagation (Supervised): When a human provides a click, they are essentially providing the correct value for a specific pixel. The authors use EM to propagate these fixed values to all other units in the network. For instance, if a user indicates a pixel belongs to a dog, the probabilistic framework spreads that semantic information to all similar pixels.
- Corrective Self-Attention (CSA) Layer: They design a specific network head that accepts human clicks as additional input channels, allowing the model to integrate feedback more effectively than standard architectures.
Datasets and Evaluation
The models were trained on the LVIS dataset (a large-scale vocabulary instance segmentation dataset). For evaluation in the interactive setting, they used two classic benchmarks:
- GrabCut: A standard dataset for interactive foreground/background segmentation.
- Berkeley: A widely used dataset for evaluating boundary detection and segmentation.
Key Insights
- Attention is a GMM: The realization that attention is a constrained Gaussian Mixture Model (GMM) moves it from a heuristic trick to a well-understood statistical framework.
- Explaining Away: Key adaptation allows the model to "explain away" variations in the input data. By adjusting keys to match queries, the model becomes more confident in its internal representations for a specific image.
- The "Low vs. High" Feedback Split: Different probabilistic updates help at different stages. Key adaptation provides a massive boost when the user has provided very few clicks. Value propagation makes the model much more responsive and accurate as the user provides more detailed feedback.
Key Results
Replacing standard attention with probabilistic attention in a BoTNet50 (BottleNeck Transformer) backbone led to consistently better performance across both the GrabCut and Berkeley benchmarks.
- Performance Boost: In the low feedback regime (0-3 clicks), key adaptation improved the mean Intersection over Union (mIoU) by approximately 10% compared to standard models.
- Responsiveness: In the high feedback regime, value propagation allowed the model to reach higher accuracy ceilings faster than traditional convolutional or standard attention-based heads.
Limitations and Future Work
The authors noted a specific technical hurdle: while value propagation worked excellently in the "width" block of their axial attention layers, it was less effective in the "height" block. They hypothesize this is due to the difficulty of learning high-dimensional fixed parameters in that specific configuration.
Future work could involve extending this probabilistic framework beyond image segmentation to other domains like natural language processing or video tracking, where online information needs to be integrated into a Transformer model in real-time.
Summary updated