TECHNOLOGIES

Level 2 – Neural Radiance Fields (NeRF)

Neural Radiance Fields (NeRF) provide implicit 3D-based scene representations, created from sparse 2D views of a scene, and enable the synthesis of novel viewpoints. They give the ability to implicitly parameterize scenes in the 3D domain from a collection of 2D images and will facilitate the creation of XR content. More specifically, the XReco users will be able to create novel, three-dimensional content from a collection of photos retrieved from heterogeneous sources within the NMR.

Level 2 – Relighting / Delighting

Creating content form multiple external sources has to face the challenge of merging images taken at different times of the day, with different weather and lighting conditions. The NeRF services implemented in XReco will be able to analyse such differences and compensate them when different media content is selected to be used in the generation of new XR media. The light equalizing will be then exploited to provide coherent assets to the other XR services for the generation of 3D assets.

Level 2 – 3D Reconstruction

3D modelling, reconstruction and rendering are crucial tasks for the XReco services. The 3D reconstruction methods implemented will be capable of creating multi-textured 3D triangle meshes from unorganised sets of images retrieved from the XReco NMR by analysing the correspondences between feature points detected in those images to create newly reconstructed 3D assets.

Level 2.5 – Holoportation (live – non linked to the NMR)

Going beyond the concept of 2D based videoconferencing, holoportation aims at bringing a real feeling of presence in remote immersive experiences. Users are captured as volumes and remotely projected directly into an XR scenario in real time.
Within XReco, the holoportation system will be used to provide a fully volumetric representation of humans to be instantly immersed in an XR environment such as a broadcasting TV studio with the possibility of giving an unlimited number of viewpoints. The broadcasting director can move a virtual camera around a holoported journalist or reporter with a cheap and easy to reproduce production system.

Level 2.5 – Free Viewpoint Video (live – non linked to the NMR)

The FVV system allows users to navigate a natural scene in an immersive fashion. It is capable of rendering in real-time the view from a virtual camera placed at any point within the captured scenario. Similarly to the holoportation, the FVV system can virtually place a journalist in a virtual 3D scene providing the required viewpoints in real-time and as a simple 2D video stream.

Level 3 – Media Broadcasting Authoring Tool

Facilitating the production workflow in the media broadcasting industry is one of the main goals of XReco. The Media Broadcasting Authoring Tool will provide an easy-to-use tool to the media industry professionals for being able to use the XReco technologies, both for search-and-retrieval and for the XR content generation, and an intuitive interface for the merging, manipulation and generation of timelines for XR based broadcasting media.

Level 3 – Tourism Authoring Tool 

Similarly, to the media industry, XReco will provide a tool for the generation of XR based experience for the tourism industry. The authoring tool, in this case, will serve as interface for the search and retrieval of content related to historical buildings, relevant historical figures, the generation of novel content thanks to the XR services and the placement of the media within XR touristic applications.