XReco, Copyright and NeRF: An Opportunity for the EU TDM Exceptions?

LEDE: Learn more about the potential role of the text and data mining (TDM) exceptions under EU copyright law and their role for 3D reconstruction via Neural Radiance Fields (NeRFs) and similar techniques. These exceptions may provide authorisations for the usage of key content for the development of XR content and experiences.

Three-dimensional (3D) digital assets are essential to virtual spaces of various kinds. The European Commission wants to ensure virtual worlds are adequately addressed and has set itself the target of ensuring European leadership in the virtual (VR) and augmented reality (AR) industries, the technologies enabling certain virtual worlds. AR and VR (collectively extended reality or XR) require the production of 3D digital assets, raising unique legal questions.

In this context, it is important to recall the EU’s copyright legislation, which may implicate certain practices of 3D reconstruction. This is essential in order to understand whether it is fit for the purpose of ensuring actors in the EU are able to contribute to and benefit from XR. The following zooms into the role potentially played by the text and data mining (TDM) exceptions in the EU copyright law regarding NeRFs and similar techniques.

A NeRF of Basilica of Superga generated by consortium partner RAI.

“3D digital asset” can mean any form of digital representation of an intangible asset with 3D spatial properties. 3D digital assets can come in a variety of forms. For instance, the content of common formats for 3D models, such as a rendering of a building, are 3D digital assets. Other variations of digital representations that are usable for 3D graphical applications, such as NeRFs, are also covered.

In recent years, NeRF has emerged as a technique that generates an instance-specific optimisation of an artificial neural network of the object which is based on input images of an underlying physical object (some technical and visual examples are accessible here). NeRFs can synthesise wholly new viewpoints of an object. More recently, Gaussian splatting techniques have also emerged as a technique for high-quality radiance field training.

These techniques can use even a sparse amount of input images to estimate what a potential viewpoint would look like. These techniques have been applied in various ways and continue to be further developed, including via the XReco project.

Using inputs to create new 3D digital assets triggers certain copyright issues. Copyright, as a legal framework affecting a vast range of digital content, has also attempted to continually address novel technologies and practices. Some of the more recent changes to EU copyright law may be especially relevant for forms of 3D digital asset production described above.

A key issue is whether the 3D digital asset interferes with the rights of a potential rightsholder of input images. The Digital Single Market Directive (CDSM) of 2019 introduced, among other things, two new exceptions in Artt. 3 and 4 for “text and data mining” (TDM) to the right of reproduction, the right of extraction for sui generis databases and for the right of press publishers under Art. 15(1) of the CDSM. These exceptions provide that, given that the beneficiary has lawful access to the subject matter, and the specific requirements of each provision are fulfilled, they can perform TDM. TDM is defined as “any automated analytical technique aimed at analysing text and data in digital form in order to generate information which includes but is not limited to patterns, trends and correlations” (Art. 2(2) CDSM).

Techniques such as NeRF or Gaussian splatting can satisfy this definition of TDM. Virtually any neural network structure is captured as an “automated analytical technique” in the sense of the CDSM, provided that it analyses information in digital form: the input images. The generated information can also be represented in different ways, including which can further be exported in a variety of formats such as a mesh or point cloud. This means that where a person has lawful access to images, subject to the requirements of the provisions, such reconstruction methods would fit this exception.

There are, of course, crucial caveats to this:

• It is not always clear if images are covered by copyright or the applicable related rights. Photographs, for instance, are protectable subject-matter under Art. 6 of the Term Directive where “they are the author’s own intellectual creation”, however, “other photographs” are not harmonised across the Member States. This means that input images may not be covered by the rights to which TDM is an exception in the first place.
• The prerequisite of “lawful access” and the so-called “opt-out” under Art. 4(3) CDSM may limit the availability of the copyright exceptions significantly (Margoni and Kretschmer). This is particularly likely where private ordering and technological protection measures are involved. The opt-out is not available vis-à-vis research organisations and cultural heritage institutions under Art. 3, yet the prerequisite of lawful access remains.
• The scope of the rights of reproduction and extraction themselves may be an issue. Demonstratively, the wording of the InfoSoc Directive is broad, covering acts that are “direct or indirect, temporary or permanent reproduction by any means and in any form, in whole or in part”. A key question is thus whether the acts performed by the techniques of 3D reconstruction qualify as acts of “reproduction”.
• The so-called “three-step test”. This test, emerging from international copyright law, attenuates the scope and reach of all exceptions and limitations to copyright law. It applies also to the TDM exceptions via Article 7(2) of the CDSM Directive in conjunction with Article 5(5) of the InfoSoc Directive. The uncertainty this test creates for the EU framework of exceptions has already been extensively discussed by Senftleben (2004, 2010).

The above has highlighted a key issue that emerges from the potential increased usage of 3D reconstruction techniques such as NeRFs or Gaussian splatting to generate 3D digital assets. There are legal questions unaddressed here, including those emerging from EU copyright law, that are further discussed in the first XReco Deliverable in Work Package 3 (D3.1).

This blog is based on a blog published by the KU Leuven Centre for IT and IP Law (CiTiP).