The FIFA World Cup is about to kick off on Thursday, but researchers have developed an AI that can project a 3D soccer game right on your kitchen table.
Researchers from the University of Washington, Facebook and Google have created the first end-to-end, deep learning system that transforms a YouTube video of a soccer game into a moving 3D hologram.
They extracted 12,000 2D images of players from FIFA video games, and then trained a convolutional neural network on 3D player data taken from the soccer video games.
Researchers from the University of Washington, Facebook and Google have created the first end-to-end, deep learning system that transforms a YouTube video of a soccer game into a moving 3D hologram
Recreating something as dynamic as a soccer match isn’t easy, the authors explain.
‘There are numerous challenges in monocular reconstruction of a soccer game,’ the researchers wrote in a new study describing the work.
‘We must estimate the camera pose relative to the field, detect and track each of the players, re-construct their body shapes and poses, and render the combined reconstruction.’
The system was able to accurately predict depth maps and combine them with color footage to reconstruct each player in 3D.
‘It turns out that while playing Electronic Arts FIFA games and intercepting the calls between the game engine and the GPU [Graphics Processing Unit], it is possible to extract depth maps from video game frames. In particular, we use RenderDoc to intercept the calls between the game engine and the GPU,’ the team stated.
‘FIFA, similar to most games, uses deferred shading during gameplay. Having access to the GPU calls enables capture of the depth and color buffers per frame. Once depth and color are captured for a given frame, we process it to extract the players.’
However, the ball is still a work in progress.
‘We present the first end-to-end system that accomplishes this goal (short of reconstructing the ball, which remains future work),’ they wrote.
The researchers used Microsoft’s HoloLens smart glasses to test their creation. They superimposed the match on a virtual soccer pitch, and it can be watched from a variety of angles using a 3D viewer or an Augmented Reality (AR) device anywhere in the world
The system was able to accurately predict depth maps and combine them with color footage to reconstruct each player in 3D
They superimposed the match on a virtual soccer pitch, and it can be watched from a variety of angles using a 3D viewer or an Augmented Reality (AR) device anywhere in the world.
The researchers used Microsoft’s HoloLens smart glasses to test their creation.
They projected ten high-resolution professional soccer games found on YouTube, even though the system was only trained on synthetic video game footage.
However, in a real-world scenario, the system delivered excellent results.
They will reveal their research at the at the annual Computer Vision and Pattern Recognition (CVPR) conference June 18 – 22 in Salt Lake City, Utah. For now, more of it can be see on YouTube.