Talking Avatar Visits XReco’s Virtual Studio

At XReco, we offer various templates for your creations – for example, a virtual news studio. You can use it to connect your own tools and assets to it, such as a talking avatar! This is exactly what we did in our crossover experiment with EU project SERMAS. Read all about how it works and why using an avatar in a virtual newsroom can be handy.

Gathering information while being aware of unpredictable or sensitive surroundings – for some, this is professional life. Being a journalist can be dangerous. To provide aide, the EU-funded SERMAS project has created an avatar called “Guardia” for XR security training for journalists, and we have integrated the 3D security agent into our XReco virtual studio.

It is a crossover experiment between XReco and SERMAS, which stands for “Socially-acceptable Extended Reality Models And Systems”. Guardia’s “day job” is to support journalistic trainers and guide journalists interactively through different levels of safety training in an XR environment. That way, places can be shown that are otherwise difficult to access – something our XReco 3D reconstruction services also offer to a certain extent.

Figure 1: SERMAS Avatar “Guardia” in XReco’s Virtual Studio.

Now, Guardia has had a day off to look around our virtual studio. This allows us to demonstrate that animated 3D objects, such as talking and moving avatars, can also be implemented in the XReco authoring tool. In other words, this experiment proves that you can bring your own assets, including animated avatars, that were not produced with XReco technologies, and integrate them in, for instance, our Virtual Studio Template in Unity.

SERMAS provided the avatar, XReco the virtual news studio. Putting them together didn’t take much time, just some tinkering. We worked with our XR Authoring Tool, an adapted, easier-to-use version of the game engine Unity.

The glTF version (Graphics Language Transmission Format) of the avatar was easy to use with the help of a plugin that is already integrated in the XReco Authoring Tool. Customizing the camera settings and angles was quickly done as well; Unity beginners could do it too. The audio recording, or rather, voice of the avatar, was AI-generated by us (with elevenlabs) and also swiftly connected to Guardia.

Once we had implemented the animation setup, everything worked smoothly. However, you may need some practice to synchronize the avatar’s mouth movements and voice audio file.

If you want to use our virtual studio to place a chatbot or an LLM (Large Language Model) in it, i.e., an avatar that can “answer” in real time and has a text chat function, this will work too. For instance, the option to integrate ChatGPT using a ready-made, online plugin is available. This can then be used in our authoring tool but does not come from the authoring tool itself. However, it means simple keyboard input from the user can be passed on to ChatGPT and the answers can be presented by an avatar. It can’t be done in five minutes, but it is certainly possible.

Here is the result:

With our project crossover, we have shown that external tools and technologies, such as avatars and their animations, can be connected to and used with existing XReco technology – just like with FVV (Free Viewpoint Video System).

Integrating an avatar into a virtual news studio can have various potential benefits for newsmakers:

• Quick adaptation of the avatar to the target group, including appearance and multilingualism: an avatar can recite the same text in different languages without much effort.
• This goes hand in hand with: Cost savings.
• New possibilities for anonymization: Journalists who want to remain anonymous or do not want to be seen can use the avatar in the virtual studio instead.
• Protagonists or eyewitnesses of a story can also be distorted with the help of avatars, like in this ZDF documentary, where a neo-Nazi dropout opens up about his past. To protect his identity, a virtual person was created and given a new voice.
• As part of an AR presentation (in a virtual studio or any other virtual environment), an avatar can explain topics using 3D models that appear next to it. This would allow showing scenarios that are hard to depict live.
• People who are no longer alive could be interviewed as avatars. For instance, XReco partner RAI has created a 3D model of Maria Callas.

Deutsche Welle (DW) is a German public international broadcaster that produces news in 32 languages. Within XReco, DW is responsible for project coordination and is a use case partner creating news media demonstrators.

Authors of this article are Olivia Stracke and Nicolas Patz. Olivia is part of XReco’s News Media Demonstrator team, has worked as a freelance journalist in the past, and wrote her master’s thesis on VR journalism. Nico is one of XReco’s two project coordinators and works as a project manager with a knack for storytelling at DW Innovation.

Never miss an update on the latest developments regarding avatars in virtual productions.