Convolutional Networks as Extremely Small Foundation Models: Visual Prompting and Theoretical Perspective

Jianqiao WangniarXiv 2024 · DOI · arXiv

Abstract

This paper introduces SDForest (Semi-parametric Deep Forest), a "visual prompting" framework that treats pre-trained convolutional networks as small foundation models for video object segmentation (VOS). By combining frozen deep features with a simple, learning-free ensemble of Random Forests and Guided Filters, the authors achieve competitive, real-time performance on a CPU without any task-specific fine-tuning. Theoretical analysis via Rademacher complexity supports the superior generalization of this low-complexity prompting module.

Introduction

In the current "GPT era," large language models are adapted to new tasks via prompting or light fine-tuning. However, in computer vision—specifically Video Object Segmentation—the dominant trend remains end-to-end training of massive, over-parameterized models. These models are computationally expensive and often struggle with generalization when faced with real-world data distributions different from their training sets.

This research addresses the fundamental trade-off between model complexity and generalization error. Using the principle of Occam’s Razor, the paper asks if high-quality segmentation can be achieved by "prompting" a generic pre-trained model with the first frame of a video, rather than performing heavy backpropagation. The goal is a method that is "instant-learning," mathematically interpretable, and efficient enough to run on standard hardware.

Methodology

The proposed solution is the Semi-parametric Deep Forest (SDForest). It breaks the segmentation task into a pipeline that avoids end-to-end retraining:

  1. Feature Extraction: The system uses a frozen, pre-trained backbone (EfficientNet-B0) to extract generic semantic features. These are treated as "foundation" features that remain unchanged.
  2. Visual Prompting: Instead of fine-tuning the CNN, the model "learns" the target object from the first frame using a Random Forest regressor and Logistic Regression. This ensemble acts as a prompting module, mapping deep features to object masks.
  3. Tracking: It utilizes a search window (borrowed from the SiamMask tracker) to focus the model's attention and reduce computational overhead.
  4. Refinement: To ensure spatial consistency, the model uses SLIC superpixels for soft mean pooling and an Image-Guided Filter (IGF) to snap coarse deep-feature predictions to high-resolution edges.
Overview of the SDForest pipeline, illustrating the integration of deep features, tracking, and refinement modules.

Theoretical Perspective

The author provides a theoretical justification using VC dimension and Rademacher complexity, arguing that because this prompting module is significantly simpler (lower complexity) than a full deep network, it is theoretically guaranteed to generalize better to unseen frames. The first-frame ground truth mask is viewed as a "visual prompt." The model doesn't need to "know" what a cat is globally; it just needs to learn the specific features of this cat in this video instantly.

Experiments and Results

The authors evaluated their approach on the DAVIS 2016 and DAVIS 2017 benchmarks. The deep backbone was pre-trained on ImageNet, and no additional training was performed on the DAVIS datasets themselves.

  • Competitive Accuracy: On DAVIS 2016, SDForest achieved a Mean J-measure (region similarity) of 73.8 and a Mean F-measure (contour accuracy) of 68.9. While slightly lower than some state-of-the-art fine-tuned models, it outperformed several deep learning baselines that require significantly more data.
  • Efficiency: The model runs at 45 FPS, making it one of the few high-performance segmentation methods capable of real-time operation without a high-end GPU.
  • Generalization: Qualitatively, the model produced highly detailed masks (e.g., clearly separating the legs of a camel) that were often more geometrically accurate than those produced by end-to-end deep models.
Qualitative comparison of estimation methods on the DAVIS 2016 dataset, showing the superior detail of the semiparametric approach.

Conclusion

SDForest demonstrates that "instant learning" via visual prompting is a viable and efficient alternative to massive end-to-end models. By combining "new school" deep learning for semantic features with "old school" non-parametric computer vision (Random Forests, superpixels, and Guided Filters), the method handles the subjective task of recognition and the objective task of edge alignment effectively. While dependent on external trackers, this paradigm offers a sustainable path for robotic tasks where on-device adaptation is critical.

Summary updated

Abstract

Comparing to deep neural networks trained for specific tasks, those foundational deep networks trained on generic datasets such as ImageNet classification, benefits from larger-scale datasets, simpler network structure and easier training techniques. In this paper, we design a prompting module which performs few-shot adaptation of generic deep networks to new tasks. Driven by learning theory, we derive prompting modules that are as simple as possible, as they generalize better under the same training error. We use a case study on video object segmentation to experiment. We give a concrete prompting module, the Semi-parametric Deep Forest (SDForest) that combines several nonparametric methods such as correlation filter, random forest, image-guided filter, with a deep network trained for ImageNet classification task. From a learning-theoretical point of view, all these models are of significantly smaller VC dimension or complexity so tend to generalize better, as long as the empirical studies show that the training error of this simple ensemble can achieve comparable results from a end-to-end trained deep network. We also propose a novel methods of analyzing the generalization under the setting of video object segmentation to make the bound tighter. In practice, SDForest has extremely low computation cost and achieves real-time even on CPU. We test on video object segmentation tasks and achieve competitive performance at DAVIS2016 and DAVIS2017 with purely deep learning approaches, without any training or fine-tuning.
BibTeX
@misc{wangni2024convolutional,
  author = {Jianqiao Wangni},
  title = {{Convolutional Networks as Extremely Small Foundation Models: Visual Prompting and Theoretical Perspective}},
  year = {2024},
  eprint = {2409.10555},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.10555},
}
Figure