Instance-Level Future Motion Estimation in a Single Image Based on Ordinal Regression and Semi-Supervised Domain Adaptation

Kyung-Rae Kim, Yeong Jun Koh, Chang-Su KimIEEE Access 2020 · DOI

Introduction

Humans possess an innate "visual common sense" that allows them to look at a static photograph and intuitively predict what will happen next. We can see a pedestrian near a curb and guess if they are about to step into the street or continue along the sidewalk. For computer vision systems, however, this task is notoriously difficult. Traditional motion estimation usually requires video sequences to calculate velocity between frames.

FM-Net predicts instance-level direction, speed, and action from a single image, utilizing scene context to distinguish between walking on a sidewalk and crossing a street.

This research introduces FM-Net, a deep learning model designed to estimate "Future Motion" (FM) from a single static image. Unlike previous methods that focused on pixel-level optical flow without object awareness, FM-Net provides instance-level predictions—specifically direction, speed, and action—for pedestrians, vehicles, and animals across diverse real-world environments.

Methodology: The FM-Net Architecture

FM-Net is built upon a DenseNet-121 backbone and incorporates three core innovations to bridge the gap between static pixels and dynamic intent:

  1. Multi-Context Pooling (MCP): The model recognizes that an object's motion is heavily influenced by its environment. MCP extracts features from two regions: a tight bounding box around the object (local features) and a larger $2h \times 2h$ area (global context). This allows the AI to factor in environmental cues like crosswalks, sidewalks, and traffic signals.
  2. Cyclic Ordinal Regression (COR): Standard classification treats all incorrect predictions as equally wrong. However, in directional estimation, guessing "North" when the target is "North-East" is a minor error compared to guessing "South." Because directions are circular (0° is equivalent to 360°), the authors implemented a cyclic regression scheme using binary classifiers to respect this unique geometry.
  3. Semi-Supervised Domain Adaptation (SSDA): To ensure the model remains robust when deployed in new cities or environments, the researchers used an adversarial training strategy. This aligns the model's internal representations by leveraging a large labeled source dataset alongside a small labeled target set and a vast amount of unlabeled target data.

The FM Dataset

To train and evaluate the model, the authors released the FM Dataset, one of the most comprehensive benchmarks for single-image motion prediction. It includes:

  • Pedestrians: 11,342 instances from CityPersons and YouTube.
  • Cars: 15,895 instances from the KITTI dataset.
  • Animals: 6,626 instances (cats, dogs, horses).

Every instance is manually annotated with one of eight directions, three speed classes (stop, slow, fast), and specific action labels such as "jaywalking" or "walking along the sidewalk."

Results and Applications

FM-Net demonstrated high accuracy, particularly in directional prediction. For pedestrians, the model achieved a 94.87% "Direction+" accuracy, which counts both exact matches and near-misses (adjacent cardinal points).

The model can be applied to crowd analysis, clustering individuals based on their predicted future motion paths in dense environments.

Beyond simple prediction, FM-Net serves as a powerful "prior" for other computer vision tasks. When integrated with a standard object tracker (MDNet), FM-Net allowed the system to focus its search on a small, fan-shaped region in the predicted direction of motion. This optimization allowed the tracker to search 33% fewer candidates, resulting in a 175% speedup (increasing from 1.13 fps to 1.98 fps) without sacrificing accuracy. Furthermore, the model proved capable of analyzing complex crowd dynamics by clustering individuals based on their predicted trajectories.

Conclusion and Future Work

While FM-Net represents a significant step forward in instance-level motion estimation, challenges remain. Some static images are inherently ambiguous—such as a person standing perfectly still versus one just beginning a stride. Additionally, severe occlusions can still degrade prediction quality. Future research will focus on estimating finer-grained, continuous motion vectors and expanding the model's ability to understand more complex human-object interactions, such as a person entering or exiting a vehicle.

Summary updated

BibTeX
@article{kim2020instance,
  author = {Kyung-Rae Kim and Yeong Jun Koh and Chang-Su Kim},
  title = {{Instance-Level Future Motion Estimation in a Single Image Based on Ordinal Regression and Semi-Supervised Domain Adaptation}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3003751},
}
Figure