Comprehensive evaluation of tracking systems by non-photorealistic simulation

Christine Dubreu, Antoine Manzanera, Eric BohainProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2007 · DOI

TL;DR

This paper introduces a novel method for evaluating object tracking algorithms by generating non-photorealistic, yet statistically representative, synthetic video sequences. This approach bypasses the need for expensive photorealistic simulators and difficult-to-obtain real-world datasets with ground truth, enabling more comprehensive and efficient performance assessment.

What Problem Does This Paper Address?

Evaluating object tracking algorithms is crucial for understanding their reliability and robustness across diverse operating conditions (e.g., varying weather, backgrounds, and target characteristics). The traditional method of using real video sequences with manually generated ground truth is problematic due to the immense effort required for ground truth creation and the difficulty in obtaining a dataset that covers the full spectrum of operational scenarios. While photorealistic simulation offers a solution, it demands costly advanced platforms, making it inaccessible to many researchers. This paper aims to bridge this gap by proposing a simpler, yet effective, method for generating synthetic datasets that are statistically representative of real-world conditions.

Methods

The core of the proposed method lies in translating operational criteria of tracking systems into objective, formal parameters. These parameters are then used to synthesize non-photorealistic video sequences. The process involves:

  • Discriminating Perceptual Parameters: Identifying key static parameters (size, shape, color, contrast, texture features like coarseness, directionality, and regularity) that characterize objects and backgrounds, independent of their motion or illumination.
  • Static Scene Synthesis: Generating synthetic texture fields based on these static parameters. This involves a sequential procedure starting with a basic texture (e.g., a Gaussian function), stretching it to achieve desired coarseness and directionality, adding noise and matching histograms, and finally using a Markov Random Fields-based algorithm for a less artificial appearance.
  • Discriminating Dynamic Parameters: Defining parameters for object and background motion, including rotation, translation, and scale.
  • Dynamic Scene Synthesis: Applying the dynamic parameters to the static target and background models to create a sequence of frames. The background is symmetrized and replicated to avoid re-synthesizing pixels.
  • Database Synthesis: Generating a large dataset by varying these parameters to cover a wide range of operating conditions.

Datasets

The paper validates its approach by comparing tracking algorithm performance on:

  • Real Sequences: Infrared and visible sequences from naval, airborne, and ground cameras, as well as sequences from the VIVID database. Ground truth for these sequences was manually generated.
  • Synthetic Sequences: Newly generated non-photorealistic sequences created using the proposed method, with parameters extracted from the corresponding real sequences.

The paper does not introduce a new benchmark dataset but rather a method for generating synthetic datasets that can be used for evaluation.

Key Insights and Ideas

  • Statistical Representativeness over Photorealism: The key insight is that for evaluating low-level tracking algorithms, statistical representativeness of operational conditions is more critical than photorealistic visual fidelity.
  • Parameterization of Operational Criteria: The ability to translate complex operational criteria (like "varying lighting" or "fast motion") into a manageable set of formal parameters is crucial for generating meaningful synthetic data.
  • Validity Domain Determination: The synthetic approach allows for precise control over parameters, enabling researchers to systematically determine the "validity domain" of a tracking algorithm – the range of conditions under which it performs acceptably.
  • Cost-Effective Evaluation: This method offers a significantly more accessible and cost-effective alternative to traditional evaluation methods relying on expensive simulators or extensive manual annotation of real data.

Key Results

The paper demonstrates that tracking algorithms evaluated on their synthetic sequences yield performance metrics (specifically, the standard deviation of the distance to the ground truth) consistent with their performance on corresponding real sequences. For example, the table in Section 3.3 shows comparable standard deviations for correlation and centroid algorithms on both real and synthetic pairs. The paper also presents figures illustrating the robustness of two tracking algorithms (correlation and centroid) to changes in contrast, target displacement, and target rotation, showing that the synthetic approach can quantitatively reveal these characteristics.

Limitations and Future Work

  • Limitations:
    • The current method is primarily suitable for low-level object tracking algorithms. It does not handle complex scenarios like static or dynamic occlusions, or highly complex target and background textures.
  • The generated textures are described as "artificial" and "simple."
  • Future Work:
    • The authors suggest that the method can be extended to simulate more elaborate scenes by increasing the number of discriminating parameters. This could involve simulating complex textures from multiple sources and controlling static/dynamic occlusions by generating multiple targets.
  • This would enable more accurate evaluation and circumscription of algorithm validity domains, albeit at the cost of more input parameters.

Summary updated

Abstract

As more and more research effort is drawn into object tracking algorithms, the ability to assess the performance of these algorithms quantitatively has become a fundamental issue in computer vision. Because tracking systems have to operate in widely varying conditions (different weather conditions, background and target characteristics, etc), a large test bed of video sequences is needed in order to obtain a comprehensive evaluation of a tracker across the whole range of its operating conditions. However, it is very unlikely that a dataset of real video sequences representative of the whole range of operating conditions of a tracker together with its ground truth could be obtained, and building a realistic synthetic dataset of such sequences would require costly advanced simulation platforms. In the new evaluation method proposed in this paper, the operational criteria of the tracking system are turned into objective measures and used to generate a synthetic dataset, non-photorealistic, but statistically representative of the whole range of operating conditions. The assessment of an algorithm using our method provides both a quantitative evaluation of the algorithm and the borders of its validity domain. The performance measurement of an algorithm on a synthetic sequence is shown to be consistent with the measurement on a real sequence with the same criteria. The benefit of this approach is twofold: it provides the developer with a way to concentrate on the weaknesses of his algorithm, and helps the system designer to choose the algorithm that best fits the operating constraints.
BibTeX
@inproceedings{dubreu2007comprehensive,
  author = {Christine Dubreu and Antoine Manzanera and Eric Bohain},
  title = {{Comprehensive evaluation of tracking systems by non-photorealistic simulation}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2007},
  doi = {10.1117/12.720821},
}

Loading related content...

Figure