How is XReco being built?

XReco is an innovative ecosystem that provides media creators with new possibilities in XR content production and media data management. To achieve this, we have utilized a range of advanced technologies, which you can learn more about on this page.

Level 1

Level 1 – Neural Media Repository (NMR)

The NMR is a modern, AI powered, content sharing platform. The involved neural technology will enable search & retrieval of media content based on the analysis of visual appearance , geometrical characteristics and correspondences between heterogeneous representations of (three-dimensional) XR content.

The NMR will provide search and retrieval over media and XR content generated both within the XReco platform and held in external repositories. In particular, the novel content, generated from the retrieved media and thanks to the XReco services, will be made available again in the NMR and offered via a marketplace.

Level 1 – Rights and Monetization Management

Generating novel content from multiple sources, especially in an AI enabled automatic (or semiautomatic) manner, will imply crucial challenges in terms of the definition of ownership and monetization rights. The XReco services will be able to select partially, or totally, previously generated media and use, merge and update them according to the decision taken by the AI implementations involved in each XR technology.
The Right and Monetization Management system implemented within XReco and based on Blockchain technology will take into account a specifically designed ownership modelling, assigning the correct, and proportional IP rights to the content providers who own the sources used in each generation process.

Level 2

Level 2 – Neural Radiance Fields (NeRF)

Neural Radiance Fields (NeRF) provide implicit 3D-based scene representations, created from sparse 2D views of a scene, and enable the synthesis of novel viewpoints. They give the ability to implicitly parameterize scenes in the 3D domain from a collection of 2D images and will facilitate the creation of XR content. More specifically, the XReco users will be able to create novel, three-dimensional content from a collection of photos retrieved from heterogeneous sources within the NMR.

Level 2 – Relighting / Delighting

Creating content form multiple external sources has to face the challenge of merging images taken at different times of the day, with different weather and lighting conditions. The NeRF services implemented in XReco will be able to analyse such differences and compensate them when different media content is selected to be used in the generation of new XR media. The light equalizing will be then exploited to provide coherent assets to the other XR services for the generation of 3D assets.

Level 2 – 3D Reconstruction

3D modelling, reconstruction and rendering are crucial tasks for the XReco services. The 3D reconstruction methods implemented will be capable of creating multi-textured 3D triangle meshes from unorganised sets of images retrieved from the XReco NMR by analysing the correspondences between feature points detected in those images to create newly reconstructed 3D assets.

Level 2.5 – Holoportation (live – non linked to the NMR)

Going beyond the concept of 2D based videoconferencing, holoportation aims at bringing a real feeling of presence in remote immersive experiences. Users are captured as volumes and remotely projected directly into an XR scenario in real time.
Within XReco, the holoportation system will be used to provide a fully volumetric representation of humans to be instantly immersed in an XR environment such as a broadcasting TV studio with the possibility of giving an unlimited number of viewpoints. The broadcasting director can move a virtual camera around a holoported journalist or reporter with a cheap and easy to reproduce production system.

Level 2.5 – Free Viewpoint Video (live – non linked to the NMR)

The FVV system allows users to navigate a natural scene in an immersive fashion. It is capable of rendering in real-time the view from a virtual camera placed at any point within the captured scenario. Similarly to the holoportation, the FVV system can virtually place a journalist in a virtual 3D scene providing the required viewpoints in real-time and as a simple 2D video stream.

Level 3

Level 3 – Media Broadcasting Authoring Tool

Facilitating the production workflow in the media broadcasting industry is one of the main goals of XReco. The Media Broadcasting Authoring Tool will provide an easy-to-use tool to the media industry professionals for being able to use the XReco technologies, both for search-and-retrieval and for the XR content generation, and an intuitive interface for the merging, manipulation and generation of timelines for XR based broadcasting media.

Level 3 – Tourism Authoring Tool 

Similarly, to the media industry, XReco will provide a tool for the generation of XR based experience for the tourism industry. The authoring tool, in this case, will serve as interface for the search and retrieval of content related to historical buildings, relevant historical figures, the generation of novel content thanks to the XR services and the placement of the media within XR touristic applications.