Action Recognition From Skeleton Data via Analogical Generalization Over Qualitative Representations

Kezhen Chen, Kenneth D. ForbusAAAI 2018 · DOI

Introduction

Most modern action recognition systems rely on deep learning and complex statistical features. While highly accurate, these models are often "black boxes"—they can identify that a person is performing an action like "waving," but they cannot explain the underlying logic of that conclusion. This lack of transparency poses challenges for trust and debugging in sensitive AI applications.

Researchers at Northwestern University have proposed an alternative approach that bridges the gap between high-performance recognition and human-like explainability. By leveraging cognitive science principles of relational representations and analogy, the system treats human motion as a series of qualitative "sketches." This allows the AI to recognize actions based on structural similarities rather than just numerical patterns.

Methodology: A Four-Stage Pipeline

The proposed system moves from raw sensor data to abstract reasoning through a structured pipeline. This process ensures that the final classification is based on meaningful human-interpretable relationships.

The four-stage action recognition pipeline, moving from raw Kinect data to final classification.
  1. Action Segmentation: Raw 3D Kinect data is projected into 2D "front" and "right" views. The system segments the motion of key body parts (such as hands and hips) into intervals where the direction of movement is qualitatively uniform.
  2. Relational Enrichment: These segments are converted into "sketch graphs." Using CogSketch, the system automatically adds qualitative spatial relations to these snapshots, such as whether a hand is above the head or if an arm is bending.
  3. Action Generalization: The system employs the Sequential Analogical Generalization Engine (SAGE) to compare multiple examples of the same action. By merging these examples, it creates a probabilistic "schema" that captures the essence of the action while filtering out individual outliers.
  4. Classification via Retrieval: To classify a new action, the system uses MAC/FAC (a two-stage analogical retrieval model) to find the most similar schema in its learned library.

Qualitative Representations and Dynamic Feature Selection

A core innovation of this research is the move from quantitative coordinates to qualitative states. By ignoring precise XYZ coordinates and focusing on states like "moving up" or "stationary," the system becomes robust to differences in body size and minor execution variations.

To manage these representations, the authors implemented Dynamic Feature Selection. This process automatically determines which qualitative relations are most useful for a specific dataset, significantly improving accuracy over manual feature selection.

Table of qualitative relations used to describe body states and movements.

As shown in the table above, the system uses specific descriptors like bodyStatus (bending vs. straight) and handPosition (raised vs. put down) to build its internal model of an action. These descriptors provide the "vocabulary" for the system's explanations.

Experimental Results

The model was evaluated on three public Kinect-based benchmarks, demonstrating that symbolic, relational AI can compete with statistical methods:

  • Florence 3D Action Dataset: Achieved 74.2% accuracy using dynamic feature selection.
  • UTD-MHAD: Achieved 80.3% accuracy on a subset of 19 actions. The system struggled slightly when actions were qualitatively identical, such as distinguishing between "swipe right" and "draw triangle."
  • UTKinect-Action3D Dataset: Achieved an overall accuracy of 88.5%, with perfect (100%) performance on distinct actions like "walk," "pick up," and "wave."

Conclusion and Future Work

The primary advantage of this analogical approach is its inherent explainability. Because the system uses Structure Mapping Engine (SME) logic, it can generate a visual "explanation sketch" showing exactly which parts of a test motion correspond to a learned action schema.

While the system currently faces some information loss due to its high level of abstraction—sometimes failing to distinguish between similar actions like "drinking" and "answering a phone"—future work aims to incorporate more complex dynamic encoding and "near-miss" learning to refine its schemas. This research suggests that human-inspired, relational AI remains a vital path toward creating transparent and trustworthy intelligent systems.

Summary updated

BibTeX
@inproceedings{chen2018action,
  author = {Kezhen Chen and Kenneth D. Forbus},
  title = {{Action Recognition From Skeleton Data via Analogical Generalization Over Qualitative Representations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11328},
}
Figure