KD-Judge: A Knowledge-Driven Automated Judge Framework for Functional Fitness Movements on Edge Devices

Shaibal Saha, Fan Li, Yunge Li, Arun Iyengar, Lucas Alves, Lanyu XuarXiv 2026 · DOI · arXiv

TL;DR

KD-Judge is a novel, knowledge-driven framework that automates repetition (rep) judging in functional fitness by using LLMs to translate unstructured rulebooks into deterministic, machine-readable constraints. By combining a Retrieval-Augmented Generation (RAG) pipeline with an intelligent dual-strategy caching mechanism, the system delivers transparent, rule-grounded rep validation faster than real-time on edge devices like the Jetson AGX Xavier.

What Problem Does This Paper Address?

In functional fitness competitions (like CrossFit), consistently enforcing movement standards is a massive challenge. Human judges face intense cognitive load, needing to make split-second, subjective decisions on complex, high-frequency movements. To complicate matters, competition rules frequently evolve from year to year.

While AI has been applied to sports analytics before, existing methods typically rely on black-box deep learning models or reference-based comparisons (comparing an athlete to a video of a professional instructor). These approaches lack transparency, fail to provide auditable step-by-step decisions, and cannot easily adapt when a rulebook changes. Furthermore, running heavy computer vision and pose estimation models frame-by-frame creates a severe computational bottleneck on resource-constrained edge devices.

Methods

KD-Judge solves these issues using a three-stage, knowledge-driven architecture that decouples rule interpretation from execution:

  1. Stage 1: Unstructured to Semi-Structured: The system uses a resource-efficient RAG pipeline powered by a quantized embedding model and GPT-4.1 to ingest official rulebooks (e.g., International Functional Fitness Federation and CrossFit manuals) and extract movement-specific text into a semi-structured JSON format.
  2. Stage 2: Semi-Structured to Structured: Using Chain-of-Thought (CoT) prompting, the semi-structured rules are mapped onto a standardized, pose-based keypoint schema (e.g., converting "hips and knees fully extended" into explicit geometric constraints like Angle(hip, knee, ankle) ≈ 180°). A human-in-the-loop step verifies these rules to eliminate biomechanical redundancies.
  3. Stage 3: Deterministic Judge Engine: The structured rules are executed by a rule-based validation engine using 2D pose estimation (via MMPose) and person tracking.
Fig. 2: Overview of the proposed KD-Judge framework. (Top) Stages 1–2 generate structured rules from unstructured stan- dards. (Bottom) Stage 3 employs a rule-based judging system to determine rep validity and rep boundaries.

To make edge deployment viable, the authors introduced a Dual Strategy Caching Mechanism:

  • Detector Caching (DC): Reuses the person bounding box across consecutive frames rather than re-running the object detector every single frame.
  • ROI-based Temporal Caching (RTC): Computes pixel-wise differences between consecutive region-of-interest (ROI) patches. If the visual change is below a calibrated threshold, it skips pose inference entirely and reuses the previous frame's keypoints.

Datasets

The framework was evaluated using two datasets:

  • Custom Calibration Dataset: A dataset generated using GPT-5 (and validated via DeepSeek R1 and human experts) consisting of context-query pairs from fitness rulebooks, used specifically to calibrate the RAG retrieval thresholds.
  • CFRep Dataset: An open-access benchmark for functional fitness repetition validation containing 64 videos of squats, deadlifts, and double unders filmed from front, side, and diagonal angles. The authors extended this dataset with fine-grained, rep-level start and end timestamps to evaluate temporal alignment.

Key Insights and Ideas

  • Rule Adaptability via Decoupling: Because the core judging engine reads rules from an LLM-generated schema rather than having them hard-coded, the system can adapt to entirely new competition rules simply by swapping out the input text document.
  • Exploiting Temporal Redundancy: Video frames of an athlete performing exercises are highly redundant. The authors realized that minor visual changes mean stable motion, allowing them to skip expensive pose estimation steps safely.
  • Viewpoint Sensitivity: The choice of camera angle heavily impacts rule validation. Front views excel at capturing vertical center-of-mass shifts (like squats), whereas side views are mandatory for depth-dependent rules (like back inclination in deadlifts) due to perspective foreshortening.

Key Results

  • Faster-than-Real-Time Performance: KD-Judge consistently achieved a Real-Time Factor (RTF) of less than 1.0, meaning it processes video faster than the actual playback speed.
  • Massive Edge Speedups: When tested on the resource-constrained Nvidia Jetson AGX Xavier, the combined caching strategy delivered up to a 3.36× speedup for pre-recorded videos and an astonishing 15.91× speedup in live-streaming scenarios compared to the non-caching baseline.
  • High Rule Fidelity: In generation quality benchmarks, GPT-4.1 achieved the highest Mean Weighted Score (0.92) for accurately extracting and structuring rules without introducing hallucinations.

Limitations and Future Work

While highly effective, the current evaluation of KD-Judge is limited to the three movement types available in the CFRep dataset (squats, deadlifts, and double unders). The authors also noted that extremely fast movements, such as the rapid arm rotations in double unders, can be difficult to capture accurately at standard 30 frames-per-second video rates. Future work will focus on expanding the framework to support a wider diversity of movements and scaling up the evaluation datasets.

Summary updated

Abstract

Functional fitness movements are widely used in training, competition, and health-oriented exercise programs, yet consistently enforcing repetition (rep) standards remains challenging due to subjective human judgment, time constraints, and evolving rules. Existing AI-based approaches mainly rely on learned scoring or reference-based comparisons and lack explicit rule-based approaches, limiting transparency and deterministic rep-level validation. To address these limitations, we propose KD-Judge, a novel knowledge-driven automated judging framework for functional fitness movements. It converts unstructured rulebook standards into executable, machine-readable representations using an LLM-based retrieval-augmented generation and chain-of-thought rule-structuring pipeline. The structured rules are then incorporated by a deterministic rule-based judging system with pose-guided kinematic reasoning to assess rep validity and temporal boundaries. To improve efficiency on edge devices, including a high-performance desktop and the resource-constrained Jetson AGX Xavier, we introduce a dual strategy caching mechanism that can be selectively applied to reduce redundant and unnecessary computation. Experiments demonstrate reliable rule-structuring performance and accurate rep-level assessment, with judgment evaluation conducted on the CFRep dataset, achieving faster-than-real-time execution (real-time factor (RTF) < 1). When the proposed caching strategy is enabled, the system achieves up to 3.36x and 15.91x speedups on resource-constrained edge device compared to the non-caching baseline for pre-recorded and live-streaming scenarios, respectively. These results show that KD-Judge enables transparent, efficient, and scalable rule-grounded rep-level analysis that can complement human judging in practice.
BibTeX
@misc{saha2026kd,
  author = {Shaibal Saha and Fan Li and Yunge Li and Arun Iyengar and Lucas Alves and Lanyu Xu},
  title = {{KD-Judge: A Knowledge-Driven Automated Judge Framework for Functional Fitness Movements on Edge Devices}},
  year = {2026},
  eprint = {2604.19834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.19834},
}

Loading related content...

Figure