As experienced connoisseurs of the VR/AR and mixed reality scene Sebastian Zwez and I, Kevin Proesel, have seen a lot but one thing remains: the Volumetric Capturing Studio from Microsoft. A studio where it is possible to record actors in 3D to generate 3D models of them that can later be used in any VR/AR/MR application.
Since we have been at the forefront of developments in experiential content and virtual reality for the Serviceplan Group for several years, such an addendum of possibilities is incredibly interesting for us and we absolutely had to have a look at it. Accordingly, we were delighted by the invitation to the Microsoft Mixed Reality Academy in Redmond and could hardly wait to visit the Microsoft Studios located there. A true Disneyland for mixed reality disciples like us.
While I create digital scenography for brands and products for Saint Elmo's Group and my team works on holographic developments in the areas of mixed reality in retail communication campaigns, Sebastian is responsible for the AR/VR/360 area in the Plan.net Innovation Studio, where the innovative strength of the agency groups is impressively presented. They are equipped with the latest devices from HTC VIVE to Meta 2 and Microsoft HoloLens. The Innovation Studio can be visited at the Haus der Kommunikation in Munich.
What strikes you as soon as you walk on the Microsoft Campus – it's far more than a campus: The extensive area northeast of Seattle extends over various locations and is so large that it was naive of us Germans to believe that you could walk around the campus. That was lesson number one. In America, everything is one size bigger. For this reason, Microsoft maintains its own shuttle network so that employees here can get from A to B.
Lesson number two: The Germans are on time, but in the wrong place at first. It quickly becomes clear that Microsoft today is more than Windows and Office. In recent years, the company has reinvented itself and put its concentrated energy into innovative aspects of a comprehensive, cross-platform service offering. This became clear that morning when we are standing in front of the wrong door on the giant campus behind which we are talking about "extreme computing". Another aspect that you don't associate with Microsoft at first and that makes it clear to you that "great things are being worked on here".
After knocking on the wrong door, the gates of the long-awaited "Stage C" finally open for us. A green screen studio of considerable size lies before us, large enough for all visitors of the Mixed Reality Academy Group. Ten people stand in the center of a circularly arranged camera setup and look around curiously. So that's it, the Mecca of mixed reality, the Holy Grail of volumetric capturing.
Jason Waskey and Spencer Reynolds from the Mixed Reality Capture Stage team tell us how it all works. The cameras arranged in a circle record us simultaneously from all sides. One part of the cameras takes a flat RGB image, while the other part is able to capture a depth portrait of us. Together, this results in a mix of data that allows the depth information and the RGB texturing to be combined into 3D objects of the filmed persons in their natural movement. The result is a stream that shows people in motion, but allows viewing from all perspectives (the so-called free viewport streaming). People are recorded volumetrically – hence the definition of volumetric capturing.
The exciting thing – and here we have paid close attention to the technical experts – is that the process is more or less automated. The capturing software works through the capture and conversion in several steps. The results are of a very high quality and, we must admit without envy, the best we have seen so far in comparable production processes.*
It must be said that the technology is not per se new and Microsoft has been working here for seven years to make the process smaller, faster and more flexible by connecting it to its (own) cloud. There are various providers and clever minds who conduct research in this field and have also developed remarkable processes, all of which pursue completely different approaches, partly similar but in detail. We have dealt extensively with previous technologies and can say that the Microsoft setup is especially in the modular structure and above all quite advanced in the accuracy of the detailed recording.
The approach is impressive in the consideration of creating an open database that creates Volumetric Content for any application. Since these are 3D objects, they can easily be integrated into other projects and combined with other "classic" 3D data. This enables us to produce complete volumetric film sequences and to become part of our holographic experiences. And all this regardless of the target device or the target environment – whether AR, VR or MR. This makes the content layer more dense and relevant in the long run, since you no longer encounter stylized fake avatars, but real plastic holograms. A point that quickly becomes clear to us when we look at the material we were allowed to shoot ourselves on the capturing stage C – we are quite vivid and everyone can look at us from all sides, and we're beautiful from behind, too.
Microsoft has turned into a relevant player (once again). What's more, up to now the IT manufacturer is the only player with usable solutions in the field of mixed reality. These include Windows Mixed Reality platform and several devices with the flagship HoloLens. However, it will be exciting to see how the industry will develop with similar approaches from other major players. Apple and Google will certainly not come up with little things here in the next few years, but will take the competition seriously, which, if you believe Tim Cook, will do nothing else but revolutionize the industry in a similar way to the introduction of the iPhone.
We are very excited and looking forward to it, as the demand for 3D content will definitely increase enormously. And with it our possibilities to tell immersive stories, which open the door to a future in which completely new dimensions of storytelling have to be explored. The real reality will be dramatically expanded once again by such approaches and technologies.
We thank Microsoft for inviting us to Redmond and the great experiences in the Volumetric Capturing Studio. Special thanks to the team of Steve Sullivan (https://www.linkedin.com/in/steve-sullivan-90b557/) and Michael Gallelli as well as Thomas Heigl and Sandro Stark from the Industry Lead Team from Germany, who made sure that we could take such an impressive look into the future of the matter!
*We took a close look at the exact technical data and backgrounds, and Jason and Spencer were grilled on every last detail, but due to confidentiality, we can't go into details here.