Transfer Learning by Sharing Support Vectors

Vitaly Henry Ablavsky, Carlos Becker, Pascal FuaInfoscience 2012

Abstract

Machine Learning techniques play an increasingly vital role in the analysis of Biomedical imagery, as in all other areas of Computer Vision. However, in this specific context, they suffer from the fact that experimental conditions and protocols change often and that acquiring sufficient amounts of new training data after each image acquisition is impractical. In this paper, we propose an effective method to train a non-linear SVM using a very small amount of new data by leveraging data obtained under different conditions. Unlike earlier approaches, ours takes full advantage of the kernelized SVM formulation, does not depend on a loss function that is sensitive to outliers, and yields a quadratic optimization problem. We demonstrate its effectiveness for the purpose of classifying pixels in electron microscope image stacks and delineating linear structures in optical microscopy and retinal scans. Our method outperforms two state-of-the-art transfer-learning approaches in terms of accuracy and computational complexity.

Introduction: The Challenge of Data Scarcity

In fields like biomedical imaging, researchers face a frustrating bottleneck: machine learning models are highly sensitive to changes in experimental conditions. If a researcher changes a microscope's settings or moves from a "hippocampus" sample to a "striatum" sample, a perfectly trained model often fails to generalize.

Standard practice requires retraining the model from scratch, but annotating 3D electron microscopy stacks or retinal scans is a "painstaking effort" that can take days or months of expert labor. This paper addresses this bottleneck through Transfer Learning—leveraging an existing, well-annotated "reference" dataset to train a new model for an "input" dataset that has only a handful of manual annotations.

Methodology: Sharing Support Vectors

The proposed method builds upon the classic Support Vector Machine (SVM) framework, adapted for multi-task learning. The core innovation lies in how knowledge is transferred between tasks while maintaining computational efficiency.

Weight Decomposition and Support Vector Sharing

The authors represent the weight vector ($w_t$) for each task as a combination of a shared component ($w_0$) and a task-specific component ($v_t$). The optimization objective penalizes large task-specific components, effectively forcing the support vectors of the new task to stay "close" to those of the reference task. This allows the model to "borrow" the structural knowledge of the source task to prevent overfitting on the data-poor target task.

The Crucial Role of Per-Task Bias

Unlike previous attempts at this logic, the authors include a unique bias term ($b_t$) for each task. This is identified as a critical component for maintaining accuracy when data distributions differ significantly, as it allows the model to account for global shifts in image intensity or contrast between different datasets.

Kernelization and Efficiency

The method is fully non-linear. By deriving a revised kernel function, the model operates in high-dimensional feature spaces while remaining a standard quadratic optimization problem. This formulation is significantly faster than competing "maximum-entropy" methods; in benchmarks, the proposed method trained in approximately 20 seconds, compared to 300 seconds for state-of-the-art alternatives.

Experimental Evaluation

The paper validates the approach across several challenging biomedical domains where manual annotation is particularly costly.

Mitochondria Detection in 3D EM

The method was tested on 3D Focused Ion Beam Electron Microscopy (FIB-EM) image stacks of brain tissue (Hippocampus vs. Striatum). When the target dataset had only 1% to 2% of the labels of the reference set, the proposed method significantly outperformed independent training and other transfer learning baselines.

3D electron microscopy stack used for mitochondria segmentation in brain tissue.

Ridge Detection and Retinal Scans

The method successfully transferred knowledge from noisy filaments in brain tissue to blood vessels in retinal scans (using the DRIVE dataset). It outperformed competitors particularly in the "low-data" regime, demonstrating that the shared support vector approach is robust even when transferring knowledge between seemingly disparate biological structures.

Retinal fundus images from the DRIVE dataset used for blood vessel delineation.

Key Insights and Robustness

A major finding of the study is the importance of the hinge-loss formulation. By using a standard SVM hinge-loss rather than a Least-Squares (LS) loss, the model remains robust to outliers and label noise—common issues in medical imaging due to human error or image ambiguity. Qualitatively, the proposed approach produces much cleaner, more accurate segmentations compared to baseline models, which often suffer from high false-positive rates or missed structures when data is scarce.

Conclusion and Future Work

While the method shows significant promise, the authors note limitations in ultra-low-data scenarios (e.g., only 1 to 6 positive samples), where standard cross-validation for parameter selection becomes uninformative. In these cases, specialized model-averaging techniques may still be required. However, the underlying logic of weight decomposition and shared support vectors provides a robust roadmap for adapting kernel-based methods to the rapidly changing environments of clinical and biological research.

Summary updated

Abstract

? This work was supported in part by the ERC MicroNano project. Abstract. Machine Learning techniques play an increasingly vital role in the analysis of Biomedical imagery, as in all other areas of Computer Vision. However, in this specific context, they suffer from the fact that experimental conditions and protocols change often and that acquiring sufficient amounts of new training data after each image acquisition is impractical. In this paper, we propose an effective method to train a non-linear SVM using a very small amount of new data by leveraging data obtained under different conditions. Unlike earlier approaches, ours takes full advantage of the kernelized SVM formulation, does not depend on a loss function that is sensitive to outliers, and yields a quadratic optimization problem. We demonstrate its effectiveness for the purpose of classifying pixels in electron microscope image stacks and delineating linear structures in optical microscopy and retinal scans. Our method outperforms two state-of-the-art transfer-learning approaches in terms of accuracy and computational complexity.
BibTeX
@inproceedings{ablavsky2012transfer,
  author = {Vitaly Henry Ablavsky and Carlos Becker and Pascal Fua},
  title = {{Transfer Learning by Sharing Support Vectors}},
  booktitle = {Infoscience},
  year = {2012},
}
Figure