Automatic, objective scoring of pull-ups (and other GTO exercises) from video — pose estimation, rep counting and technique-violation detection.
Live analysis: skeleton tracking, bar & grip detection, rep counting, elbow angles and technique violations — all overlaid in real time.
Passing the GTO (“Ready for Labour and Defence”) fitness standard normally requires a human judge to watch every repetition and rule on technique — slow, subjective and hard to scale. Pull-up Detection AI replaces that manual step with computer vision: a candidate uploads a video, and the system detects their pose, counts valid repetitions, and flags technique violations against the official GTO rules — then returns an annotated video with the verdict.
It is built as a two-service system:
- Processing service — the CV engine (FastAPI). Pose estimation → tracking → per-exercise analysis → annotated video.
- Web service — a Flask platform where athletes upload attempts and trainers review results.
- 🎯 Rep counting with a verdict — separates valid, incomplete and rule-violating repetitions
- 🧍 Whole-body pose estimation — RTMPose (133 keypoints, incl. face for chin-over-bar checks)
- 🏋️ Domain detectors — trained YOLOv8 models for the bar and grip type (over/under-hand)
- 📏 Interpretable technique rules — joint angles & keypoint geometry, mapped 1-to-1 to official GTO violations
- 🎥 Annotated output video — skeleton, counters, angles and violations burned into the frame
- ⚡ Optimised pipeline — batched inference, DeepSORT tracking, Kalman-filter frame-skipping and an async writer thread (≥2× speedup)
- 🧩 Pluggable by design — abstract
ExerciseProcessor/Visualizer/Normalizermake new exercises easy to add - 🐳 One-command stack —
docker compose upbrings up both services + shared storage
Yellow = non-critical violation · Red = critical violation. Red lines mark the bar region the hands must stay within.
At the top level, a browser talks to the web service, which hands videos to the CV processing service. Both share one database and one video store (a deliberate single-node design).
Internally, the processing service is organised around a small set of reusable components. ExerciseManager receives requests, queues them and writes results; Pipeline runs the frame-by-frame loop; a per-exercise ExerciseProcessor does the actual analysis. Pose models sit behind an abstract interface, so a model can be swapped without touching exercise logic.
The two services coordinate through the shared DB and storage: the web backend uploads the raw video and creates the attempt record; the processing service picks it up, writes the annotated video back and updates the tables.
Video is processed in batches of 60 frames. Each batch flows through:
Three ideas make it fast and accurate:
- 🎞️ Batched inference. Frames are buffered and sent to the pose model together for efficient GPU/CPU use.
- 🧭 Multi-person tracking. DeepSORT assigns a stable ID to each person, so the candidate is never confused with bystanders — even as people enter and leave the frame.
- 🪄 Kalman-filter frame-skipping. The model runs only on significant frames; for skipped frames a per-keypoint Kalman filter (
[x, dx, y, dy]constant-velocity model) predicts the skeleton. It also smooths detector jitter. Because the tracked exercises are cyclic and near-linear, this cuts compute without losing the frames where a rep is actually made or missed.
Profiling showed video writing was the bottleneck starving the GPU, so writing was moved to a separate thread fed by an async queue — CPU writes while the GPU infers. Net effect: at least a 2× pipeline speedup.
Cyclic exercises share one approach: split the stream into single repetitions, then score each one.
Every repetition is modelled as a 4-state machine — hang → pull → chin-over-bar hold → descent — with transitions driven by elbow angles and chin position relative to the bar.
Detected violations, straight from the official GTO methodology, include: wrong grip, hands too wide/narrow, arms not fully extended, bent legs, feet apart, insufficient bottom-hold, uneven arm bend, and facing away from the camera. Each maps to explicit angle/geometry checks, so every verdict is interpretable.
Both reuse the same processor abstractions with their own rules — body-line straightness and elbow depth for push-ups; scapula lift and locked lower back for sit-ups (crunches). Push-ups run on RTMPose; sit-ups are at MediaPipe-research stage.
The pose backbone was chosen after a deliberate progression:
| Approach | Verdict |
|---|---|
| MediaPipe Pose | Fast & cross-platform, but unstable under motion and hard to fine-tune → research only |
| YOLOv8-pose | Fine-tuned on a custom dataset — great for grip and bar, weaker for full skeleton |
| RTMPose-m ✅ | Chosen backbone: 2.22 GFLOPS, COCO-WholeBody + UBody, 133 keypoints incl. face; top-down with an RTMDet-m detector |
Dataset. 150 open-source pull-up videos (~135 min) were labelled in CVAT: human skeleton, bar keypoints (with/without supports) and grip bounding boxes (over/under-hand) — 2 400 train / 350 val frames.
Detector metrics (validation):
| Target | Box P | Box R | Box mAP@50 | Pose P | Pose R | Pose mAP@50 |
|---|---|---|---|---|---|---|
| Bar | 0.975 | 0.527 | 0.531 | 0.979 | 0.996 | 0.991 |
| Grip | 0.899 | 0.769 | 0.778 | — | — | — |
| Skeleton | 0.942 | 0.962 | 0.991 | 0.928 | 0.945 | 0.984 |
The custom skeleton model was ultimately replaced by RTMPose; the grip detector was kept, and the bar detector is backed up by a manual bar-marking fallback in the UI (bars vary too much for one model to nail every case).
Pull-up rep counting (TP = valid, FP = false valid, TN = correct incomplete, FN = missed):
| TP | FP | TN | FN | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|
| 355 | 2 | 62 | 49 | 0.89 | 0.995 | 0.88 | 0.93 |
Per-batch timing on an RTX 2080 Ti / 4-core CPU / SSD — GPU vs CPU-only:
SQLite (single-node), designed so new exercises are just new tables. Attempts carry a status code (-2 rejected by trainer · -1 rejected by model · 0 pending · 1 approved by model · 2 approved by trainer).
A Flask app (Jinja2 templates, Flask-Login / -WTF / -Bcrypt) with two roles:
- Athlete — register, upload an attempt, see status (passed / not passed) and the annotated video
- Trainer — review every attempt, override the automatic verdict and leave feedback
CSRF-protected forms, bcrypt-hashed passwords, auto-escaped templates.
git clone https://github.com/simeonkolchin/pullup-detection-ai.git
cd pullup-detection-ai
cp .env.example .env # set SECRET_KEY, TRAINER_SECRET
# download model weights into processing-service/models/ (see below), then:
docker compose up --build- Web platform → http://localhost:8080
- Processing API (Swagger) → http://localhost:8000/docs
All four weight files are published as a GitHub Release (too large for git). Grab them with the helper script:
bash processing-service/scripts/download_weights.shOr download manually and place them at the paths referenced in .env:
| Asset | Destination (under processing-service/models/) |
|---|---|
rtmdet-m.pth — RTMDet-m person detector (public) |
mmpose/det/weights/ |
rtmpose-m.pth — RTMPose-m whole-body pose (public) |
mmpose/pose/weights/ |
bar_detection.pt — fine-tuned bar detector |
pull_ups/yolo/bar_detector/weights/ |
hand_position.pt — fine-tuned grip detector |
pull_ups/yolo/hand_detector/weights/ |
The two RTM* weights also live in the MMPose model zoo; the YOLOv8 weights are fine-tuned for this project.
pullup-detection-ai/
├── processing-service/ # CV engine (FastAPI)
│ ├── app/engine/ # ExerciseManager, Pipeline, tracker, mmpose/, yolo/
│ │ └── exercise/ # pull_ups · push-ups · sit-ups processors
│ ├── research/ # MediaPipe / MMPose / per-exercise notebooks
│ └── models/ # weights (git-ignored)
├── web-service/ # Flask web platform
│ ├── app/ # auth, video, trainer logic
│ ├── templates/ # Jinja2 pages
│ └── db/gto_db.sql # schema
├── docs/ # architecture diagrams, benchmarks, demo
└── docker-compose.yml # web + processing + shared storage
- Give each service its own database (decouple web ⇄ processing)
- Migrate SQLite → PostgreSQL and video storage → object store (horizontal scaling)
- Finish sit-ups on RTMPose; add more GTO exercises
- On-device / mobile inference with lightweight RTMPose
- Biometric candidate verification for official attempts
MIT © Simeon Kolchin