XReco brought Maria Callas to the XR/AI World marking 100 years since the birth of La Divina. Our advances in Artificial Intelligence, CGI & 3D Modeling opens up new opportunities for creative entrepreneurs supported by the novel ecosystem developed within the consortium.
Production-Ready Research Outcomes
Every day there are many research results and technological advances being developed that clearly push the boundaries of science. However, are they any good for real-life production needs? This is a question often asked by the public when looking at EU project outcomes so we decided to put ourselves to the test early on and see whether our tools are any good for such a Use-Case, namely “To bring a celebrity born 100 tears ago back to life”.
Maria Callas was an American-born Greek soprano and one of the most renowned and influential opera singers of the 20th century. To bring her back to life, we first decided to assemble novel pipeline from the multiple cutting edged components being developed to showcase their joint potential. These element included technologies like Neural Search, Neural Radiance Fields & 3D Gaussian Splatting and Stable Diffusion.
How Can You Do That?
Figure 1: Creating a model of Maria Callas using AI-based novel technologies.
As a first step Callas’ face was reconstructed from 50 Internet images with the help of AI-based head shape estimation methods, traditional view-based 3D modeling techniques. Furthermore, we trained a Generative AI Stable Diffusion model to pose and re-imagine the Diva from any angle in 4K.
Next, we also captured and virtually reconstructed the same Opera House in Budapest, where Angelina Jolie recently filmed a part of her new role about the life of the world-famous opera singer. In order to do that, we scanned the outside of the landmark building by flying a commercial drone around it, and used neural rendering (NeRF) as the representation of choice to construct a 3D scene with interactive virtual camera control. The interior of the Opera House was also scanned, but this time a set of 360º panoramic photos were take from which a high resolution 3D Gaussian Splat representation was derived.
Figure 2: 360º panoramic image of the interior of the Opera House from which the 3DGS model was reconstructed.
Virtual Production on a Dime
To bring it all together the AI-based Callas model was combined with the 3D elements in real-time within an interactive virtual production set built entirely in Unity3D. To control the head motion and expressions of the character we used GenAI driven by a CGI animated head and a digital body stand-in using ControlNet.
The process to create the 3D assets and the animated scenes cost only the fraction of the time and resources otherwise would have been needed for such a solution without these novel technologies. Thus this demo Use Case not only proved the technical viability of such solutions, but also the commercial value and potential savings offered by our approach.
As a recognition of this new approach we were invited to presented our work at the Mobile World Congress in Barcelona as part of their 4YFN program. Further details maybe found in Demo Videos and the links below.
Figure 3: Controlling the pose and expressions of virtual Maria Callas using GenAI & Stable Diffusion
Figure 4: Angeline Jolie (in a scene from her new film) turning into virtual Maria Callas (animated)
More about the production process in the video.
FFP’s Work in XReco
FFP is a creative-industry partner testing the developed XReco technologies for use in real-world productions and select Use-Cases in the Media, Travel and Entertainment sectors. FFP also advances research on NeRFs, 3D Gaussian splatting and multi-camera capture systems.
Stay tuned, the developments for NeRF and Virtual Productions continue. Never miss an update again? Sign up for the XReco newsletter.
Author:
Barnabas Takacs, Ph.D. (https://www.linkedin.com/in/barnabastakacs/ ), Head of R&D, Special Projects @FFP Productions, Vienna, AT (FFP https://www.ffp.at/ )
Resources:
- Demo Video NeRFs for Virtual Production Evaluation Visual Summary (https://www.youtube.com/watch?v=Oo1A30w3s4Y)
- NeRF technology https://xreco.eu/realistic-3d-worlds-with-neural-radiance-fields/
- Mobile World Congress, 2024 Barcelona https://www.mwcbarcelona.com/agenda/sessions/4422-ai-arias-and-digital-divas
- https://en.wikipedia.org/wiki/Maria_Callas
- Angelina Jolie Films in Budapest Opera House https://dailynewshungary.com/angelina-jolie-will-shoot-in-budapest-for-three-months/
Follow XReco!