The Problem

Elite sport is one of the last domains where performance feedback is still largely subjective. A coach watches an athlete and offers qualitative guidance — "your weight transfer is off", "you're rotating too early" — based on years of experience and trained intuition. That intuition is valuable, but it's not scalable, it's not consistent, and it can't be reviewed frame by frame.

For surfing specifically, the challenge is acute. Judging in competitive surfing is already based on objective criteria — speed, power, flow, commitment — but coaching feedback to athletes rarely is. A surfer completes a maneuver in 2-3 seconds. The biomechanics that determine whether it scores well happen faster than a coach can consciously track.

The US Olympics Committee and USA Surfing Team wanted to change that: give athletes and coaches a system that could see what the human eye misses and turn it into actionable feedback.

What I Built

The Computer Vision Pipeline

The system takes video of athletes in training and competition and runs it through a multi-stage computer vision pipeline:

1. Pose Estimation & Tracking

Using PyTorch-based models, the pipeline extracts skeletal keypoints from each frame — tracking joint positions, angles, and velocities across the athlete's body throughout a maneuver. This is the foundation everything else builds on.

2. Maneuver Segmentation

Surfing isn't a continuous motion — it's a sequence of discrete maneuvers (cutbacks, aerials, tube rides) separated by transitional paddling. The system automatically segments video into maneuver windows, so analysis is tied to meaningful athletic events rather than arbitrary time slices.

3. Biomechanical Feature Extraction

For each segmented maneuver, the pipeline extracts a feature set: center of gravity trajectory, weight distribution, rotation timing, limb angle sequences, and velocity profiles at key moments. These features capture the mechanics that experienced coaches identify as performance drivers.

4. Feedback Generation

Extracted features are compared against a reference model built from high-scoring maneuver examples. Deviations are surfaced as structured feedback — specific, timestamped, and tied to the frame where the deviation occurred. Coaches can review with the athlete frame by frame.

Working with Elite Athletes

One of the more interesting design challenges was the human side: the feedback had to be legible to athletes and coaches who think in athletic terms, not ML terms. "Your shoulder rotation peaks 0.3 seconds late relative to your hip rotation" is accurate but not useful on a beach. The output layer translates model findings into coaching language, calibrated with the USA Surfing coaching staff over several iteration cycles.

Challenges

Video variability. Ocean environments are visually noisy — changing light, moving water, spray, wetsuits. Standard pose estimation models trained on clean environments perform poorly. Significant fine-tuning was required to achieve reliable keypoint tracking in surf conditions.

Ground truth for "good." What does an optimal cutback look like? Building the reference model required working closely with coaches to label and annotate high-quality maneuver examples — a time-intensive process that had to be done carefully to avoid encoding individual coaching preferences as universal ground truth.

Latency for in-session use. Feedback that arrives the next day is less useful than feedback that arrives between sets. The pipeline was optimized to produce results within minutes of a training session ending, enabling same-session review.

Impact

Deployed with the USA Surfing Team in preparation for the Olympics
Gives athletes frame-level biomechanical feedback on maneuvers that complete in under 3 seconds
Translates computer vision output into coaching language usable directly on the beach
Featured in press coverage in January 2023