How Peloton makes use of laptop imaginative and prescient to enhance coaching

Were you unable to attend Transform 2022? Check out all Summit sessions in our on-demand library now! Look here.

While you’re doing push-ups, squats, abdominal workouts, dumbbell presses, jumping or stretching, a device on your TV accompanies you throughout your workout.

You are tracked on your form, your completion (or lack thereof) of an exercise; Get recommendations on which cardio, bodyweight, strength, or yoga workout to do next; and you can work towards achievement badges.

This is the next level home fitness experience made possible by Peloton Guide, a camera-based, TV-mounted exercise machine and system powered by computer vision, artificial intelligence (AI), advanced algorithms and synthetic data.

Sanjay Nichani, head of Peloton’s computer vision group, spoke about the development – and ongoing improvement – of the technology in a live stream at Transform 2022 this week.

AI-driven motivation

Peloton Guide’s computer vision capability tracks members and recognizes their activity, crediting them with completed moves and providing recommendations and real-time feedback. A “Self Mode” mechanism also allows users to pan and zoom their device to observe themselves on the screen and ensure they are showing the correct form.

Nichani underscored the power of measurable accountability when it comes to fitness, saying that “insight and progress are very motivating.”

Reaching the final commercial Peloton Guide product is an “iterative process,” he said. The initial goal of AI is to “boot fast” by sourcing small amounts of custom data and combining it with open-source data.

Once a model is developed and deployed, detailed analysis, assessment and telemetry are applied to continually improve the system and make “targeted improvements,” Nichani said.

The machine learning (ML) flywheel “everything starts with data,” he said. Peloton developers used real world data supplemented with “a large dose of synthetic data” and created datasets using specific nomenclature for exercises and poses combined with appropriate reference materials.

Development teams also applied pose estimation and matching, accuracy detection models, and optical flow, which Nichani called “classic computer vision techniques.”

Various attributes affecting computer vision

One of the challenges of computer vision, Nichani said, is the “wide variety of attributes that must be accounted for.”

This contains:

  • environmental attributes: background (walls, floors, furniture, windows); lighting, shadows, reflections; other people or animals in the field of view; equipment used.
  • Member Attributes: gender, skin tone, body type, fitness level and clothing.
  • geometric attributes: camera user placement; camera mounting height and tilt; Member orientation and distance from camera.

Peloton developers conducted extensive field testing to address edge cases and included a feature that “nudges” users when the camera can’t see them due to a number of factors, Nichani said.

The Bias Challenge

Fairness and inclusivity are both paramount to the process of developing AI models, Nichani said.

The first step in mitigating bias in models is to ensure the data is diverse and has enough values ​​for different attributes for training and testing, he said.

Still, he noted, “A diverse dataset alone does not guarantee unbiased systems. Bias creeps into deep learning models, even when the data is unbiased.”

Peloton’s process attributes attributes to all source data. This allows models to measure performance across “different segments of attributes” and ensure no bias is observed in models before releasing them into production, Nichani explained.

If bias is uncovered, the flywheel process and deep analysis will address it – and ideally correct it. Nichani said Peloton developers observe a fairness metric of “equal opportunity.”

That is, “for a given label and attribute, a classifier predicts that label equally for all values ​​of that attribute.”

For example, to predict whether a member would perform a crossbody curl, squat, or dumbbell swing, models were built to account for attributes of body type (underweight, average, overweight) and skin tone based on the Fitzpatrick -Classification – which while widely accepted for skin tone classification, importantly still has some limitations

Still, any challenges are far outweighed by significant opportunities, Nichani said. AI is having many implications for the home fitness space – from personalization to accountability and convenience (e.g. voice-controlled commands) to guidance and overall engagement.

Providing insights and metrics helps improve a user’s performance “and really pushes them to do more,” Nichani said. Peloton aims to provide personalized gaming experiences “so you’re not checking the clock while you train.”

Watch the full Transform 2022 conversation.

Comments are closed.