Probabilistic Information Fusion for Multi-Modal Image Segmentation

Paul B. Chou, Christopher M. Brown

Introduction

In a seminal 1987 paper from the University of Rochester, researchers presented a robust framework for "information fusion"—the process of integrating data from multiple sensors or algorithms to achieve a single, coherent interpretation. Specifically applied to image segmentation, the study demonstrates how to merge noisy, disparate visual cues into a spatially consistent map by combining hierarchical trees with Markov Random Fields (MRFs).

The Challenge of Multi-Modal Fusion

Image segmentation involves partitioning an image into meaningful regions, a task often hampered by the inherent noise and incompleteness of individual visual modules, such as edge detectors or texture analyzers. The authors identify two primary hurdles:

  1. Abstraction Levels: Different modules operate at varying levels of detail (e.g., identifying a generic "geometric edge" versus a specific "horizontal edge").
  2. Spatial Consistency: Pixels are not independent; the presence of an edge at one location strongly suggests its continuation in neighboring pixels.

Methodology: Trees and Fields

The proposed architecture addresses these challenges through a dual-layered approach.

Hierarchical Knowledge Trees

Labels are organized into a tree structure where a root "Primitive" branches into categories like "Edges" and "Regions," which further subdivide into specific types. Vision modules report evidence as likelihood ratios for specific nodes.

Figure 1: Hierarchical tree structure for organizing visual labels and propagating evidence.

A message-passing scheme, inspired by Judea Pearl’s work on belief networks, propagates this evidence throughout the tree to maintain a consistent state across all levels of abstraction.

Markov Random Fields (MRFs) and Bayesian Integration

To incorporate spatial context, the authors utilize MRFs to define "prior knowledge." For instance, the MRF can be configured to favor continuous lines while penalizing isolated noise or redundant parallel edges. The core innovation lies in mathematically integrating tree-based evidence with the MRF spatial prior. The resulting posterior distribution is a Gibbs distribution, solvable via optimization techniques like Maximum A Posteriori (MAP) estimation or Maximizer of Posterior Marginals (MPM) using a Gibbs sampler.

Experimental Results

The framework was validated using synthetic datasets consisting of overlapping rectangular patches with significant Gaussian noise.

Figure 2: Experimental results showing the progression from raw, noisy input to fused, spatially consistent segmentations.

The results demonstrate a dramatic improvement over simple Boolean logic or single-sensor analysis. The system successfully "recovers" edges that might be invisible to one sensor but present in another, while effectively filtering out inconsistent noise.

Conclusion

This probabilistic approach decouples local sensor evidence from global spatial priors, creating a modular and efficient system for image understanding. While the computational cost of MRF optimization remains a challenge, the framework's ability to handle incremental updates and provide coherent segmentations from noisy data marked a significant advancement in computer vision.

Summary updated

BibTeX
@inproceedings{chou1987probabilistic,
  author = {Paul B. Chou and Christopher M. Brown},
  title = {{Probabilistic Information Fusion for Multi-Modal Image Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1987},
}
Figure