TL;DR
Standard attribution methods often fail for structured outputs (like angles or 3D rotations) because they explain raw network coordinates rather than the final decoded prediction. By attributing through the decoder differential, this paper introduces "prediction-space" explanations that ignore irrelevant changes (like vector scaling) and significantly improve the faithfulness of feature heatmaps.
What Problem Does This Paper Address?
When a neural network predicts a gaze direction, an angle, or a 3D rotation, it doesn't output these values directly. Instead, it emits a vector of raw coordinates ($z$) which a fixed, parameter-free decoder ($\pi$) maps to the final structured prediction ($y = \pi(z)$). For example, a gaze model might output a 3D vector and then normalize it to get a direction on a sphere.
The authors identify two major failure modes in current attribution methods (like Grad-CAM or Integrated Gradients) when applied to these models:
- Radial Contamination: The decoder often discards certain information. In a normalization decoder, the length of the output vector is ignored—only the direction matters. Standard attribution methods still credit inputs that change this length, even though those inputs have zero effect on the final prediction.
- Scalarization Loss: Most attribution methods require a single scalar target. If you just sum the output coordinates, you only capture sensitivity in one specific direction, potentially missing inputs that move the prediction in other, equally important directions.
Methods
The core proposal is to attribute through the decoder differential. Formally, if the model is $x \mapsto z = f(x) \mapsto y = \pi(z)$, the explanation should use the chain rule: $D(\pi \circ f) = D\pi \cdot Df$.
For common normalization decoders, this operation reduces to a tangent projection. The authors adapt several popular attribution techniques into "tangent" or "prediction-space" variants:
- Tangent Grad-CAM / HiResCAM / LayerCAM: These project the gradients onto the tangent space of the output manifold (e.g., a sphere) before generating the heatmap.
- Angular Integrated Gradients (AIG): A variant of Integrated Gradients that integrates changes in the actual predicted angle rather than raw coordinates.
- Circular/Spherical LIME: Local surrogate models that use circular or spherical regression targets.
Datasets
The authors evaluate their methods across four distinct settings:
- Synthetic Oracles: Custom-built tasks where the ground-truth relevance is known. One region of the image controls the output direction (relevant), while another controls only the discarded magnitude (irrelevant).
- Plant (S1): A circular regression task predicting the time of day from images of plants.
- Gaze (S2): Gaze direction estimation using the MPIIGaze dataset.
- Pose (SO(3)): 3D object pose estimation using the PASCAL3D+ dataset, employing both quaternion and 6D rotation decoders.
Key Insights and Ideas
- The "Invariance" Requirement: An explanation should be "decoder-invariant." If two different raw output vectors $z$ and $2z$ map to the same prediction $y$, an input that only moves the model from $z$ toward $2z$ should receive zero credit.
- Ambient Attribution can be "Below Chance": On synthetic tasks, standard "ambient" attribution assigned up to 72% of its mass to pixels that were provably irrelevant to the prediction.
- Representation Sharing: The failure of standard methods is most acute when the network's internal representations for the "relevant" and "discarded" information are highly separated. As the pathways share more features, the "gap" between ambient and tangent attribution shrinks, though it rarely disappears.
- Scalarization is a separate bottleneck: Simply making a scalar target decoder-invariant (e.g., using the sum of a rotation matrix) isn't enough. On 3D pose tasks, the full differential significantly outperformed invariant scalar targets, proving that capturing the multi-dimensional nature of the prediction is critical.
Key Results
- Gaze Estimation: Tangent Grad-CAM produced a 44% greater prediction displacement than the ambient baseline when the top 10% of attributed pixels were deleted, indicating it found much more influential features.
- 3D Pose (6D Decoder): Using the full differential improved mean deletion displacement by 14.1° over the standard coordinate sum.
- Synthetic Tasks: Tangent methods achieved near-perfect Oracle Attribution Mass Ratios (AMR $\approx 1.0$), while ambient methods hovered near or below chance (0.5).
Limitations and Future Work
- Mixed Results for LayerCAM: While tangent variants generally helped, the results for a two-polarity tangent LayerCAM were mixed, showing gains in circular regression but no detectable difference in quaternion pose.
- Invariance $\neq$ Faithfulness: The authors warn that while decoder invariance is a necessary property for a "correct" explanation, it doesn't guarantee the explanation will be useful. An invariant method can still miss important features; it just won't highlight provably irrelevant ones.
- Complexity: Some variants (like the full differential for 6D pose) require multiple backward passes, increasing computational cost compared to simple gradient-based methods.
- Future Directions: The authors suggest extending this framework to more complex structured outputs and investigating how these "prediction-space" heatmaps can better help practitioners debug models in the wild.
Summary updated