Watching from within at Canon’s volumetric video studio

We, the audience, are used to watching from the periphery. Whether it’s an onstage performance or our favourite team on the pitch, we are, of course, rapt – but usually from a distance. “Of course,” you say. “Where else would we be?” It’s a great question, and one that Canon Inc. now has the answer to:

Everywhere.

Less than an hour outside of Tokyo, Canon’s Kawasaki office is the home of a new ‘Volumetric Video’ studio. It is here that one is able to see the incredible potential of a technology that is capable of putting the audience everywhere. In the green-screened studio, real-time footage can be captured from any angle, then used to generate a reconstruction in 3D almost instantly. Together with Canon’s Free Viewpoint Video System (which in the near future will allow viewers to control how they see the scene), it could change the way we view live action forever. With it you could watch your favourite band from any point, travelling around the stage as naturally as if you were walking. Or have the perfect vantage point of a spectacular goal without any switching of cameras, as was the case when the Free Viewpoint system was put through its paces during an international rugby match in Tokyo during 2019.

The new studio opens up a world of volumetric content in some really exciting new areas, such as TV commercials, music videos and events. Not only can it take you anywhere inside the action, in almost real-time, but the fact it can also do so in 3D means that, in the near future, viewers can use VR or AR to immerse themselves in live action entertainment. Extraordinarily, the same technology could also render live scenes as holograms. It’s a glimpse of a future that turns traditional viewing inside out, and through it the audience gains free and open access to any part of the ‘stage’ while the ‘fourth wall’ between audience and performers remains intact. It feels fresh, dynamic, natural and holds endless potential for realising new ideas and reaching new audiences.

In a year when live events were sadly limited. the first project to be held at the new Canon studio was a Covid-secure livestream event for Japanese rap duo ‘chelmico’. Directed by Jun Tamukai, rappers Rachel and Mamiko were celebrating the release of their new album ‘Maze’ and used the studio to create a ‘virtual live show’ for fans. The broadcast alternated between conventional cameras and Free Viewpoint video, and viewers on YouTube saw the pair from multiple different angles – including above and below – that transitioned between 2D and 3D. They also had a peek behind the scenes, watching the duo manipulate footage of themselves with the viewpoint controls.

This incredibly fun spectacle was created with 89 4K cameras, all network connected and ready to simultaneously capture the same scene from multiple viewpoints. The images were then used to ‘construct’ high-detail 3D spatial data, using Canon’s proprietary image processing technology. The latency from image capture to Free Viewpoint output is tiny, and it took just a couple of seconds to process Rachel, Mamiko and their DJ into a 3D space. After which, a 3D camera then took over, which was freely moved around inside the 3D space to produce the 2D video content for broadcast.

Of course, the chelmico broadcast was just three people, but the spacious studio ‘capture area’ can both accommodate multiple people and many subjects moving at speed, while still supporting a vast range of video production needs. For example, the simultaneous 60 frames per second (fps) capture of a fast ten-person dance performance can take place at the studio and, again, this creates a 3D data model of all performers as they move, from all angles. This in itself has exciting implications for theatrical and scripted content – after all, who doesn’t want to get their own perspective on their favourite shows?

Putting the viewer in the driving seat in this way may feel radical today, but this exciting tech has the potential to take many directions and give us new ways of experiencing filmed content. It’s going to be truly fascinating to see how creative brains push the envelope of this volumetric technology and, as they do with so many of Canon’s innovations, create truly transformational experiences.

Learn more about Canon’s Free Viewpoint Video System and the new Volumetric Video studio.