Welcome to the next level of technological innovation! Imagine a world where reality merges with virtuality, where real 3D scenes come to life in digital space in a way never experienced before. The VV (Volumetric Video) system we are developing within the XReco project is redefining standards: a masterful fusion of 3D modelling, reconstruction and rendering, with fascinating applications in the world of tourism and media industry.
Exploring the Future: XReco and its Innovative Volumetric Video
VV is a technology that has been around for over a dozen years in the research arena and is now ready to reach consumers in their/our everyday life. The goal of VV is to first capture a real 3D scene or object (e.g., a building) by taking 2D pictures of it, typically with normal, consumer-grade cameras, and then build some kind of synthetic 3D model that can later be rendered from freely chosen viewpoints with an astonishing level of detail and realism.
Can you imagine exploring a tourist destination from the comfort of your home, with a fidelity that rivals reality itself? Thanks to XReco’s VV system, now you can immerse yourself in the architecture of iconic cities, travel back in time to witness history in its splendour, or simply discover new wonders from the sofa of your home. Besides, in the realm of streaming media, 3D reconstruction redefines visual storytelling. With unprecedented image quality, traditional 2D content is transformed into immersive augmented reality experiences arousing curiosity and excitement.
XReco’s Cutting-Edge Technology: 3D Meshes and Neural Radiance Fields
Our VV system leverages traditional 3D mesh reconstruction techniques with the latest in NeRFs (Neural Radiance Fields) technology. In both cases, 2D images are taken with normal cameras and used to build synthetic 3D models. These images, which can also be individual frames extracted from videos, are stored in XReco’s NMR (Neural Media Repository), a state-of-the-art content exchange platform powered by artificial intelligence which drives creativity and collaboration in the digital world, and offers an unparalleled repertoire for both creators and consumers.
However, the processes involved in 3D mesh reconstruction and NeRFs are different, and so are the natures of the 3D models obtained:
- Figure 1 below illustrates XReco’s 3D mesh reconstruction process: 2D images are first individually enhanced and analysed to extract feature points, which are later matched in image pairs to yield a 3D point cloud (step 1), which in turn is meshed with triangles (step 2). Besides, the input 2D images are used as well to build a texture atlas (step 3) that is mapped onto that mesh (step 4), so the global result is a traditional textured 3D triangle mesh, which can be rendered from any viewpoint and on virtually any device.
Figure 1: 3D mesh reconstruction process
The video below elaborates a bit more on XReco’s 3D mesh reconstruction process and provides at the end some details on our improved texturing technique.
However, alternative technologies can also be used to depict 3D scenes: enter NeRFs! Unlike traditional methods that explicitly model a 3D scene using points and triangles (such as meshes), NeRFs harness neural networks to learn a continuous representation of the scene.
- As shown in Figure 2 below, given our knowledge of the positions and orientations of the real cameras that captured the initial images, we can project rays and render the scene from those specific viewpoints. Simultaneously, we aim to estimate the scene appearance and density. How is this done you ask? Rays are projected through the known camera positions, and points are sampled along them, and then fed to the neural network that represents the scene as a radiance field. The neural network learns to estimate the density and colour of each input point, so colours and densities along the rays can be aggregated via volumetric rendering, and the corresponding pixels get their final colour.
Figure 2: NeRFs rendering process
XReco’s Path to a New Era in Tourism and Media
XReco’s revolutionary VV system ushers in an era of unlimited possibilities. Our 3D mesh reconstruction and NeRFs technologies, together with our NMR, offer exceptional technical performance and exciting practical applications. Join the transformation of tourism and media, where reality and imagination merge in a way never seen before.
Stay tuned, sign up the XReco newsletter.
Authors:
UPM: D. Berjón, J. Cabrera, F. Morán, J. Usón
CERTH: A. Karakottas
Follow XReco!