Abstract
The LEAK (LEArns from classification mistaKes) framework is a plug-and-play regularization strategy designed to improve point cloud semantic segmentation by analyzing a model's own errors. By automatically clustering confused classes into "macro-groups" and enforcing hierarchical feature alignment and fairness, LEAK achieves state-of-the-art performance and faster convergence across various architectures without increasing inference-time complexity.
Introduction
Semantic segmentation—assigning a label to every point in a 3D cloud—is a cornerstone of autonomous driving and robotics. However, models often struggle with two main issues: class confusion (where visually or geometrically similar classes like "car" and "truck" are mistaken) and class imbalance (where rare classes like "person" are ignored in favor of dominant ones like "vegetation"). While hierarchical approaches can mitigate these issues, they typically require manually defined taxonomies. LEAK addresses this by allowing a model to learn its own hierarchy by analyzing its own mistakes.
Methodology
LEAK employs a three-pronged approach to self-regularization during the training phase:
- Unsupervised Hierarchy Discovery: Instead of human-defined categories, LEAK analyzes the confusion matrix of a pre-trained model. Using spectral clustering, it groups classes that are frequently mistaken for one another into "macro-classes," identifying "semantic communities" based purely on model behavior.
- Hierarchical Feature-Prototype Alignment: The model maintains representative feature vectors (prototypes) for every micro-class and macro-class. The loss function forces individual point features to align with both their specific micro-prototypes and broader macro-prototypes, structuring the latent space more logically.
- Attentive Fair Weighting: To tackle imbalance, Jain’s fairness index is integrated into the loss function. This ensures the model seeks to equalize performance across all micro-classes within a macro-group rather than just maximizing overall accuracy through common classes.
Experimental Results
The framework was evaluated on major benchmarks including SemanticKITTI, Semantic3D, S3DIS, and Pascal-VOC2012. Key findings include:
- Performance Gains: LEAK improved mean Intersection over Union (mIoU) across all tested backbones, adding +1.3% mIoU to RandLA-Net and +0.7% to Cylinder3D on SemanticKITTI.
- Rapid Convergence: The model reached 50% mIoU in just 10 epochs, achieving over 91% of its final performance significantly earlier than baseline models.
- Improved Discrimination: Qualitative analysis shows a marked reduction in "bleeding" between similar classes, such as correctly distinguishing sidewalks from roads.
Discussion and Conclusion
The core insight of LEAK is treating the confusion matrix as a roadmap for optimization. Because it is a training-time regularization strategy, it is architecture-agnostic and adds zero overhead to inference time. While the authors noted some initial prototype instability in the very first few epochs, the subsequent acceleration in convergence and accuracy makes it a robust addition to point cloud pipelines. Future work may extend this "learning from mistakes" philosophy to tasks like object detection and instance segmentation.
Summary updated