Abstract
Many problems in computer vision require dealing with sparse, unordered data in the form of point clouds. Permutation-equivariant networks have become a popular solution - they operate on individual data points with simple perceptrons and extract contextual information with global pooling. This can be achieved with a simple normalization of the feature maps, a global operation that is unaffected by the order. In this paper, we propose Attentive Context Normalization (ACN), a simple yet effective technique to build permutation-equivariant networks robust to outliers. Specifically, we show how to normalize the feature maps with weights that are estimated within the network, excluding outliers from this normalization. We use this mechanism to leverage two types of attention: local and global - by combining them, our method is able to find the essential data points in high-dimensional space in order to solve a given task. We demonstrate through extensive experiments that our approach, which we call Attentive Context Networks (ACNe), provides a significant leap in performance compared to the state-of-the-art on camera pose estimation, robust fitting, and point cloud classification under noise and outliers.
Introduction
Processing point clouds—collections of data points like 3D coordinates or image correspondences—is a fundamental challenge in computer vision because these sets have no inherent order. Permutation-equivariant networks, such as PointNet, address this by using Context Normalization (CN) to share information across points via global statistics (mean and variance). However, standard CN is highly sensitive to outliers because it relies on least-squares statistics. In real-world applications like wide-baseline stereo matching, outlier ratios can exceed 80%, causing standard normalization to fail as a few noisy points skew the global context.
Methods: Attentive Context Normalization
To overcome the sensitivity of standard normalization, we introduce Attentive Context Normalization (ACN). Instead of uniform weighting, ACN learns to assign an attention weight to each point, effectively filtering out noise during the calculation of global statistics.
Robustness through IRLS
ACN is inspired by Iteratively Re-weighted Least Squares (IRLS), a classical robust optimization technique. In IRLS, weights are iteratively adjusted to down-weight points with high residuals. ACNe implements a similar logic within a deep learning framework using two attention mechanisms:
- Local Attention: Evaluates each point individually to determine its likelihood of being an inlier.
- Global Attention: A softmax-based mechanism that assesses the relative importance of points across the entire set.
Architecture
These mechanisms are integrated into Attentive Residual Blocks (ARB). By stacking these blocks, the network progressively refines its focus. Early layers maintain a broad view, while deeper layers "zoom in" on relevant data points, mimicking an automated RANSAC-like process.
Experimental Evaluation
We evaluated ACNe across several benchmarks to test its robustness:
- Robust Line Fitting: Finding a line in 2D points with heavy noise.
- MNIST Point Clouds: 2D point representations of digits with up to 60% salt-and-pepper noise.
- ModelNet40: 3D object classification under Gaussian noise and outliers.
- Photo-Tourism: Real-world camera pose estimation in challenging outdoor environments.
Key Insights and Results
ACNe demonstrates that normalization serves as a primary communication channel between points. By embedding robust statistical principles directly into the architecture, the network becomes inherently resistant to noise.
Performance Highlights
- Extreme Noise Handling: In point classification with 60% outliers, ACNe achieved 93.7% accuracy, compared to PointNet's 54.8%.
- Camera Pose Estimation: ACNe improved pose estimation accuracy by 14.1% over OANet on outdoor datasets.
- Robust Fitting: ACNe successfully fits lines even with 90% random outliers, a scenario where standard Context Normalization (CNe) fails completely.
Conclusion and Future Work
ACNe provides a significant performance leap for permutation-equivariant learning in noisy environments. While highly effective, future work could explore more complex parametric ways to blend local and global attention while maintaining equivariance. Additionally, applying Attentive Normalization to standard CNNs for handling spatial occlusions or outliers in image data remains a promising research direction.
Source code: https://github.com/vcg-uvic/acne
Summary updated