TL;DR
KD-Judge is a novel, knowledge-driven framework that automates repetition (rep) judging in functional fitness by using LLMs to translate unstructured rulebooks into deterministic, machine-readable constraints. By combining a Retrieval-Augmented Generation (RAG) pipeline with an intelligent dual-strategy caching mechanism, the system delivers transparent, rule-grounded rep validation faster than real-time on edge devices like the Jetson AGX Xavier.
What Problem Does This Paper Address?
In functional fitness competitions (like CrossFit), consistently enforcing movement standards is a massive challenge. Human judges face intense cognitive load, needing to make split-second, subjective decisions on complex, high-frequency movements. To complicate matters, competition rules frequently evolve from year to year.
While AI has been applied to sports analytics before, existing methods typically rely on black-box deep learning models or reference-based comparisons (comparing an athlete to a video of a professional instructor). These approaches lack transparency, fail to provide auditable step-by-step decisions, and cannot easily adapt when a rulebook changes. Furthermore, running heavy computer vision and pose estimation models frame-by-frame creates a severe computational bottleneck on resource-constrained edge devices.
Methods
KD-Judge solves these issues using a three-stage, knowledge-driven architecture that decouples rule interpretation from execution:
- Stage 1: Unstructured to Semi-Structured: The system uses a resource-efficient RAG pipeline powered by a quantized embedding model and GPT-4.1 to ingest official rulebooks (e.g., International Functional Fitness Federation and CrossFit manuals) and extract movement-specific text into a semi-structured JSON format.
- Stage 2: Semi-Structured to Structured: Using Chain-of-Thought (CoT) prompting, the semi-structured rules are mapped onto a standardized, pose-based keypoint schema (e.g., converting "hips and knees fully extended" into explicit geometric constraints like
Angle(hip, knee, ankle) ≈ 180°). A human-in-the-loop step verifies these rules to eliminate biomechanical redundancies. - Stage 3: Deterministic Judge Engine: The structured rules are executed by a rule-based validation engine using 2D pose estimation (via MMPose) and person tracking.
To make edge deployment viable, the authors introduced a Dual Strategy Caching Mechanism:
- Detector Caching (DC): Reuses the person bounding box across consecutive frames rather than re-running the object detector every single frame.
- ROI-based Temporal Caching (RTC): Computes pixel-wise differences between consecutive region-of-interest (ROI) patches. If the visual change is below a calibrated threshold, it skips pose inference entirely and reuses the previous frame's keypoints.
Datasets
The framework was evaluated using two datasets:
- Custom Calibration Dataset: A dataset generated using GPT-5 (and validated via DeepSeek R1 and human experts) consisting of context-query pairs from fitness rulebooks, used specifically to calibrate the RAG retrieval thresholds.
- CFRep Dataset: An open-access benchmark for functional fitness repetition validation containing 64 videos of squats, deadlifts, and double unders filmed from front, side, and diagonal angles. The authors extended this dataset with fine-grained, rep-level start and end timestamps to evaluate temporal alignment.
Key Insights and Ideas
- Rule Adaptability via Decoupling: Because the core judging engine reads rules from an LLM-generated schema rather than having them hard-coded, the system can adapt to entirely new competition rules simply by swapping out the input text document.
- Exploiting Temporal Redundancy: Video frames of an athlete performing exercises are highly redundant. The authors realized that minor visual changes mean stable motion, allowing them to skip expensive pose estimation steps safely.
- Viewpoint Sensitivity: The choice of camera angle heavily impacts rule validation. Front views excel at capturing vertical center-of-mass shifts (like squats), whereas side views are mandatory for depth-dependent rules (like back inclination in deadlifts) due to perspective foreshortening.
Key Results
- Faster-than-Real-Time Performance: KD-Judge consistently achieved a Real-Time Factor (RTF) of less than 1.0, meaning it processes video faster than the actual playback speed.
- Massive Edge Speedups: When tested on the resource-constrained Nvidia Jetson AGX Xavier, the combined caching strategy delivered up to a 3.36× speedup for pre-recorded videos and an astonishing 15.91× speedup in live-streaming scenarios compared to the non-caching baseline.
- High Rule Fidelity: In generation quality benchmarks, GPT-4.1 achieved the highest Mean Weighted Score (0.92) for accurately extracting and structuring rules without introducing hallucinations.
Limitations and Future Work
While highly effective, the current evaluation of KD-Judge is limited to the three movement types available in the CFRep dataset (squats, deadlifts, and double unders). The authors also noted that extremely fast movements, such as the rapid arm rotations in double unders, can be difficult to capture accurately at standard 30 frames-per-second video rates. Future work will focus on expanding the framework to support a wider diversity of movements and scaling up the evaluation datasets.
Summary updated