Learning to detect and localize many objects from few examples

Bastien Moysset, Christopher Kermorvant, Christian WolfarXiv 2016 · OpenReview

Abstract

The current trend in object detection and localization is to learn predictions with high capacity deep neural networks trained on a very large amount of annotated data and using a high amount of processing power. In this work, we propose a new neural model which directly predicts bounding box coordinates. The particularity of our contribution lies in the local computations of predictions with a new form of local parameter sharing which keeps the overall amount of trainable parameters low. Key components of the model are spatial 2D-LSTM recurrent layers which convey contextual information between the regions of the image. We show that this model is more powerful than the state of the art in applications where training data is not as abundant as in the classical configuration of natural images and Imagenet/Pascal VOC tasks. We particularly target the detection of text in document images, but our method is not limited to this setting. The proposed model also facilitates the detection of many objects in a single image and can deal with inputs of variable sizes without resizing.

The Challenge of Dense Object Detection

Modern object detection frameworks like YOLO, SSD, and Faster R-CNN have revolutionized computer vision but remain significantly constrained by their data requirements and architectural rigidity. These models typically rely on massive pre-training (e.g., ImageNet) and struggle with specific scenarios:

  1. Dense, Small Objects: Detecting hundreds of items, such as text lines in a scanned document, often exceeds the capacity of standard spatial grids.
  2. Variable Input Dimensions: Standard architectures often require resizing or cropping, which can distort small features or lose context.
  3. Data Scarcity: In specialized domains like historical document analysis, large-scale annotated datasets are rarely available.

Standard models often fail when multiple small objects fall into a single grid cell. The authors address these limitations by proposing a model designed for high parameter efficiency and dense localization.

Architectural Innovation: 2D-LSTMs and Local Parameter Sharing

The proposed model shifts away from the parameter-heavy fully connected (FC) layers common in detection heads. Instead, it utilizes two key innovations:

  • Local Feature Sharing via 1x1 Convolutions: By replacing FC layers with 1x1 convolutions, the model maintains spatial structure and drastically reduces parameter counts. In a comparable setup to MultiBox or YOLO, this reduces the final layer's parameters from approximately 15.6 million to just 3,700, significantly mitigating the risk of overfitting on small datasets.
  • Spatial 2D-LSTM Context Layers: To compensate for the limited receptive field of local convolutions, the authors integrate Multi-Dimensional LSTM layers. These 2D-LSTMs sweep across feature maps in four directions, providing the local predictors with global image context.
Comparison of standard CNN architectures (a, b) with the proposed fully convolutional model using 2D-LSTM layers and 1x1 convolutions (c).

Additionally, the model employs Global Bipartite Matching during training. Using the Hungarian algorithm to match predictions to ground truth allows the network to bypass the "one object per grid cell" limitation, enabling the detection of hundreds of objects per image.

Experimental Setup and Datasets

The model was evaluated on the Maurdor dataset, a complex collection of document images featuring mixed languages (French, English, Arabic) and formats (handwritten and printed). A subset of 3,995 training and 616 test pages was used, focusing on high-confidence ground-truth line positions.

Results and Key Insights

The proposed architecture demonstrated significant improvements over existing baselines:

  • Detection Performance: On the Maurdor test set, the model achieved an F-Measure of 73.8% (at IoU=0.3), substantially outperforming the optimized MultiBox baseline (48.7%) and traditional image processing methods (~40%).
  • Ablation Study: The necessity of spatial context was confirmed when removing the 2D-LSTM layers caused the F-measure to drop from 73.8% to 49.9%.
  • Versatility: The model successfully localized text across diverse document types, including maps and floor plans, where standard grid-based matching often fails.
Qualitative results showing text line detection across diverse document types including passports, maps, and floor plans.

Conclusion

By combining local parameter sharing with spatial recurrent layers, this research provides a robust solution for dense object detection in data-scarce environments. The model's ability to handle variable input sizes and hundreds of objects per image makes it particularly suited for document analysis. Future work involves integrating these localization capabilities with attention-based OCR for improved end-to-end recognition.

Summary updated

Abstract

The current trend in object detection and localization is to learn predictions with high capacity deep neural networks trained on a very large amount of annotated data and using a high amount of processing power. In this work, we propose a new neural model which directly predicts bounding box coordinates. The particularity of our contribution lies in the local computations of predictions with a new form of local parameter sharing which keeps the overall amount of trainable parameters low. Key components of the model are spatial 2D-LSTM recurrent layers which convey contextual information between the regions of the image. We show that this model is more powerful than the state of the art in applications where training data is not as abundant as in the classical configuration of natural images and Imagenet/Pascal VOC tasks. We particularly target the detection of text in document images, but our method is not limited to this setting. The proposed model also facilitates the detection of many objects in a single image and can deal with inputs of variable sizes without resizing.
BibTeX
@misc{moysset2016learning,
  author = {Bastien Moysset and Christopher Kermorvant and Christian Wolf},
  title = {{Learning to detect and localize many objects from few examples}},
  year = {2016},
  eprint = {1611.05664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.05664},
}
Figure