Pixels to Skeletons: Towards Internet-Scale Biomechanics

The analysis of human performance and health is traditionally confined to laboratory settings, limiting analysis at scale and "in the wild". From an standard RGB image or a video, can we look inside the body to gain useful information about human health? I will describe our recent work on two fronts. The first focuses on inferring the internal structure of the body, including the bones, lean tissue, and adipose tissue from a single image of a person. To make progress, we construct novel datasets that pair external 3D shape with MRI and CT data of the inside of the body and train neural networks to predict the inside from the outside. The second focuses on moving motion capture out of the lab and into the world. I will show our latest work on accurately recovering 3D human kinematics from video. We estimate both the human's motion and the motion of the camera, enabling us to infer human movement in a global coordinate system with state-of-the-art accuracy.