TL;DR
The authors introduce PLUM, a training-free method for out-of-distribution (OOD) object detection that treats the decision as a hypothesis test conditioned on bounding box geometry. By comparing detected objects only to in-distribution (ID) examples of similar size and image-border contact, PLUM achieves state-of-the-art performance without requiring retraining or external models.
What Problem Does This Paper Address?
Deep learning object detectors are notoriously overconfident. When they encounter an object from a class they weren't trained on (an OOD object), they often draw a box around it and assign it a known (ID) class with high confidence. This is a major safety concern for real-world deployments like autonomous driving.
Existing OOD detection methods often struggle because the "score" used to identify OOD objects is confounded by the bounding box's geometry. For example, a very small object or one partially cut off by the image border naturally yields different feature statistics than a large, centered one. Standard methods that apply a single global threshold across all detections suffer from "conditional bias," leading to uneven false-alarm rates—small ID objects might be wrongly flagged as OOD, while large OOD objects might slip through.
Methods
The authors propose PLUM (Pooled, Leveled, Unit-norm Mahalanobis score), which reframes OOD detection as a stratified hypothesis test.
- Stratification: Instead of a one-size-fits-all threshold, PLUM groups detections into "strata" based on their size level (from the feature pyramid) and whether they touch the image border.
- Feature Extraction at the "Task Boundary": PLUM pools features from the last backbone stage (e.g.,
res5in a ResNet-50) rather than the final classification head. The logic is that the final layers are trained to discard any information not strictly necessary for identifying ID classes, potentially throwing away the very features that distinguish an unknown object. - Mahalanobis Distance: It calculates the distance of a pooled, normalized feature to the nearest ID class mean, using a covariance matrix estimated from the training data.
- Leveling: Within each stratum, it further removes size-related trends by fitting a linear regression to the scores in log-area space.
- Ranking: Finally, it ranks a test box's score against a reference set of ID boxes from the same stratum. This provides a distribution-free bound on the false-alarm rate for every type of box.
Datasets
The method was evaluated using standard OOD object detection benchmarks:
- In-Distribution (ID): PASCAL-VOC and BDD100K.
- Out-of-Distribution (OOD): Images from MS-COCO and OpenImages that contain no objects from the ID classes.
- Detectors: Faster R-CNN (ResNet-50 FPN) and Deformable DETR.
Key Insights and Ideas
- Geometry as a Confounder: The "null distribution" (the scores of ID objects) is not static; it shifts with box size and position. Comparing a small box to a large box is an "apples-to-oranges" comparison.
- The Task Boundary: There is a specific point in a network where it stops being a general feature extractor and starts being a specialized ID classifier. Reading features before this boundary is critical for OOD detection.
- Training-Free is Competitive: PLUM outperforms many methods that require complex retraining or synthetic outlier generation, proving that better statistical handling of existing features can be more effective than adding more data.
- Stratification Improves AUROC: As shown in Figure 1, simply accounting for box size can jump AUROC from 71 to 92 in a toy example by evening out the false-alarm rates.
Key Results
PLUM achieved the highest AUROC among compared methods that do not use external models (like CLIP):
- PASCAL-VOC: Reached 91.5 (MS-COCO OOD) and 94.5 (OpenImages OOD) AUROC.
- BDD100K: Reached 95.1 (MS-COCO OOD) and 97.5 (OpenImages OOD) AUROC.
- Qualitative Success: PLUM correctly identifies OOD objects (like a zebra or a pizza) that standard thresholding might miss, while correctly keeping small or border-touching ID objects (like small airplanes or traffic signs) that standard methods might falsely reject.
Limitations and Future Work
- Reference Set Dependency: PLUM requires a set of ID detections (from the training or validation set) to serve as a reference for ranking. While the authors show this is robust, the quality of the OOD detection is tied to how well these references represent the test-time ID distribution.
- Backbone Reliance: Since it reads backbone features, its power is limited by what the backbone has learned. If a backbone is completely "blind" to certain visual patterns, PLUM cannot recover them.
- Future Directions: The authors suggest extending this "box-conditional" logic to other tasks like instance segmentation (using masks) or 3D object detection. They also note that multiple testing corrections could be applied to control the error rate over all detections in an image.
Summary updated