TL;DR
This paper introduces Global Sum Pooling (GSP), a novel pooling operation that addresses the limitations of Global Average Pooling (GAP) for object counting tasks, especially with small datasets of high-resolution images. GSP enables models to generalize better to variable input resolutions and reduces overfitting, leading to improved counting performance across multiple datasets.
What Problem Does This Paper Address?
Object counting, while seemingly simple, presents significant challenges when dealing with datasets that have a small number of high-resolution images. Traditional approaches often rely on complex architectures designed for object detection or segmentation, which require detailed annotations and can be computationally expensive. Simpler "one-look" regression models are more suitable, but they struggle with the characteristics of these datasets:
- High-resolution, variable-shaped images: Training on full images can lead to overfitting due to the small dataset size. Resizing or warping images can distort object sizes and make counting harder.
- Patch-based training: A common workaround is to train on smaller, randomly cropped patches. However, this introduces issues like partial object instances at patch boundaries and "patchwise cancellation," where overestimates and underestimates from different patches randomly cancel each other out, masking underlying errors.
- Overfitting: With limited training data, models can easily memorize specific images rather than learning generalizable features.
The paper aims to develop a model that can be trained on small patches (to mitigate overfitting) but still infer counts accurately on full-resolution images of varying sizes, overcoming the limitations of existing methods.
Methods
The core contribution of this paper is the Global Sum Pooling (GSP) operation. It's proposed as a replacement for Global Average Pooling (GAP) or fully connected (FC) layers at the end of a convolutional neural network (CNN) for object counting.
- GSP vs. GAP: While computationally similar to GAP (both involve aggregation over spatial dimensions), GSP uses summation instead of averaging. The authors argue that averaging in GAP leads to a loss of information when dealing with variable input sizes, as it averages out spatial feature responses. GSP, by summing these responses, allows the model to learn a linear mapping that scales proportionally with the number of objects and the input image size.
- Architecture: The proposed architecture uses a standard convolutional front-end (specifically, a pre-trained VGG16) followed by the GSP layer. This GSP layer outputs a fixed-size vector (e.g., 512 dimensions), which is then fed into a linear layer to produce the final scalar count.
- Training Strategy: The GSP models can be trained on smaller, randomly cropped patches from the high-resolution images. This strategy helps reduce overfitting and encourages the network to learn more localized features, acting as a "weak object instance detector" even with weak labels (scalar counts).
Datasets
The paper evaluates the proposed GSP models on four diverse datasets:
- CARPK [15]: Overhead view of car parks.
- ShanghaiTech [44]: Crowd counting dataset with images from web and streets.
- COWC [24]: Overhead view of cars in residential areas and highways.
- Wheat-Spike [17]: Overhead view of mature wheat plants.
These datasets vary in resolution, object density, and the nature of annotations (bounding boxes for CARPK, dot annotations for others). Notably, CARPK, COWC, and Wheat-Spike datasets are characterized by high-resolution and variable-sized images, making them suitable for testing the generalization capabilities of GSP.
Key Insights and Ideas
- Patchwise cancellation is a critical flaw of GAP for counting: The paper convincingly demonstrates that while GAP might appear to perform well on aggregated patch counts, the underlying per-patch errors (overestimates and underestimates) are significant and are masked by random cancellation. This makes GAP unreliable for robust counting.
- GSP learns a linear mapping: The core idea is that GSP enables the CNN to learn a direct, linear relationship between the input image features and the object count, regardless of the input image's spatial dimensions. This is a crucial generalization capability.
- Training on small patches improves localization: Even though GSP can infer on full images, training on smaller patches forces the model to focus on local features, leading to better activation maps that are more concentrated on object regions. This is a beneficial side-effect for object detection-like tasks.
- Simplicity and effectiveness: The GSP approach is presented as a simple modification to existing CNN architectures, requiring minimal changes to achieve significant performance gains.
Key Results
The GSP models consistently outperform state-of-the-art approaches across all four evaluated datasets.
- CARPK: GSP models trained on 128x128 and 224x224 patches achieved significantly lower MAE and RMSE compared to GAP models and other existing methods. For instance, GSP-224 achieved an MAE of 5.46 and RMSE of 8.09.
- ShanghaiTech (Parts A & B): GSP models demonstrated superior performance, with GSP-224 (Part A) achieving an MAE of 70.69 and GSP-128 (Part B) achieving an MAE of 9.13, outperforming methods like MCNN and CS-CNN.
- COWC: GSP-64 achieved an MAE of 11.15 and RMSE of 23.61, comparable to or better than previous methods like ResCeption.
- Wheat-Spike: GSP-96 achieved the best performance with an MAE of 80.00 (8% of average count), highlighting GSP's effectiveness even with very limited high-resolution training data.
- Qualitative results: Activation maps (CAMs) show that GSP models, especially when trained on smaller patches, produce more localized and uniform activations over object regions compared to GAP models.
Limitations and Future Work
- Optimal patch size dependency: The paper notes that the optimal patch size for training GSP models can depend on the dataset's characteristics, such as object resolution and density. For CARPK, smaller patches (64x64, 96x96) did not perform as well, attributed to issues with object instances being partially cut at patch edges when bounding boxes are shrunk for disambiguation.
- Future Work: The authors suggest that GSP could be extended to other computer vision tasks like classification and object detection. They believe GSP's ability to utilize features from image sub-regions across multiple spatial scales could offer advantages over GAP or FC layers in these domains, potentially eliminating the need for fixed-resolution input images.
Summary updated