Volumetric video – A Bootcamp

So, what is volumetric video?

It’s a technique to record a live video performance and play it back from any angle. Cameras capture multiple vantage points simultaneously, and post-processing converts all the video captures into an animated volumetric dataset and 3d model. This 3d representation can be imported into any 3d environment extending the creative potential to playback and manipulate this animated data. It brings a new dimension to working with video over a flat 2d video taken from a single camera’s perspective

But why? We can only see from one point of view.

Volumetric video data offers another level of control and flexibility. Data can be embedded into immersive real-time virtual or augmented reality applications or used as part of video productions. Volumetric video adds an engaging advantage to creative processes. Viewing 3d video on the fly while it’s playing back from any angle is a powerful proposition for creative users. You can manipulate the lighting and colour, and, with a little interpolation, AI can help clean or smooth volumetric data and enhance the overall result.

Why would I need to do that?

Although volumetric video is in its infancy, volumetric data is not a new concept. Many consumer devices have been able to record and track video playback via cameras and infrared devices in applications to interpret gesture control or scan objects and faces for pseudo-3d or, when done professionally, highly detailed 3d models and complete environmental renditions. For example, the consumer photogrammetry approach typically uses a camera or video to capture an object placed centrally, and pictures are taken from multiple angles. Take enough photographs of the subject a detailed 3d model will materialise. It is a common technique that has been available for decades. Volumetric video capture records a complete performance, and there are many use cases.

Imagine being at a sports game or a music festival as a spectator. You can replay the event and watch it from any position. It’s possible to feel like you are standing on stage or viewing a replay of a memorable sporting victory from any angle. You control where you are standing by not having physical limitations, which you can’t get to see watching television. It takes you closer to the feeling of being there.

In creative industries, you can manipulate an actor’s performance even after a recording has finished or use the data to drive other CG characters.

Volumetric video and the extrapolated 3d data enhance the degree of control over precisely what you want to achieve. It opens up many creative benefits.

The concepts are similar to other virtual production tools but where volumetric video differs is that it captures physical movements and environments, not just stationary-based objects. But both have their uses and complement each other.

When you say perspective…

I mean exactly that. The great thing about volumetric video is that it records the scene in full 3d, so you aren’t even limited to the perspectives of the capturing devices. You can move, using a 3d camera, anywhere in the scene and view it from any angle. Think of a football game, where you want to view the action from a freely movable camera. Clearly, you can have thousands of cameras, covering every available angle and distance. Well, volumetric video builds that scene in 3d, so if you want to watch a glorious goal you can do so from any point of view, up close or taking in the whole pitch (or anything in between).

One more trick of this technology is that you can put yourself in the position of any of the performers, or even a static object in the scene, for an immersive experience.

There are also clear benefits beyond the entertainment field. Imagine a medical setting where a tricky situation can be observed from different perspectives or natural world events that can be preserved digitally. The opportunities are still growing.

Right, so what else?

Think about a trip to your favourite online shop. Imagine being able to choose an outfit and see it from any angle, at your pace and your direction. Some tools already exist that simulate this, as well as placing objects virtually in a space, but volume video takes this to a whole new level. Pick a wardrobe and decorate a space, then share that with whoever you so wish, for feedback or just to show off your style.

Ok, I’m sold. How do I get started?

To start the process, you’ll want a set of cameras, positioned similarly to how motion capture works, with a mesh of cameras surrounding a volume. The difference here is that shape, light, and colour are captured, not just the motion. Because of this, an evenly lit volume might be best but that will depend on the situation.

Next up, the captured footage will need to be processed ready for the consumer, or further edited or integrated into a 3d scene. Tools like Depthkit, HoloCap and Holosuite by Arcturus are ideal for this. The latter also offers non-linear editing tools to ease your projects along.

Once processed and edited, you can begin integrating the volumetric files into augmented or virtual reality projects, deliver online with real-time navigation of your scenes and much more.

Contact Us

To find out more, call us on +353 (0)1 526 6717 or email sales@bluegfx.eu