Immersive live experiences: 8K Video, 3D audio and Beethoven’s 9th

By Merit Ariane Stephanos

As part of the Avant-Premiere Music and Media Market 2024’s Innovation Day, Mauricio Alvarez Mesa, CEO of Spin Digital, gave a special presentation entitled Immersive live experiences: 8K Video, 3D audio and Beethoven’s 9th.

The catalyst for the presentation was a live stream in 8K Video and 3D audio of Beethoven’s 9th Symphony, performed by the NHK Symphony Orchestra on December 26th 2023 at the NHK Concert Hall in Tokyo. In this collaboration between Spin Digital and NHK technologies, 8K video (7680×4320 pixels at 60 frames per second) and 3D audio (22.2 channels with MPEG-H 3D Audio) were used to create an immersive viewing and three-dimensional sound experience, with the aim of giving the viewer the impression of sitting in the best seat of the concert hall.

The concert was streamed live over the Internet to viewing locations in Tokyo, Hiroshima, and Berlin, where setups ranged from formal seating to viewers sitting on bean bags. It was filmed with a single static camera from the audience’s perspective, allowing the viewer to choose where to put their focus. The 8K video provided a wide panoramic angle with the entire orchestra and choir in focus giving a clear sense of depth with a 3D-like effect.

Each viewing location was able to use their own video and audio set up and, thanks to the capabilities of the encoding, streaming and playback software, it was possible to adapt the the 8K HEVC video and 22.2 channel MPEG-H 3D audio to different outputs, including:

• NHK Technologies Lab, Shibuya Tokyo: 8K projection, 22.2 channel 3D audio.
• Astrodesign, Tokyo: 8K 3D projection, stereo audio.
• Higashi-Hiroshima Arts and Culture Center, Hiroshima: 8K projection, 5.1 surround sound.
• Spin Digital, Berlin: 8K TV, 5.1.4 channel 3D audio.

The live stream of this concert was designed on the one hand to test the technical possibilities of streaming high quality immersive concert experiences live over the internet and, on the other, to create shared social and artistic experiences. Looking ahead, there remains the question of how we can use the 8K video and 3D audio technology to go beyond trying to reproduce the experience of being in a concert hall.

For this concert, Spin Digital provided the software for 8K live and low-latency HEVC encoding, streaming over the internet, and media playback with MPEG-H 3D audio. The live stream of Beethoven’s 9th Symphony is part of a long collaboration between Spin Digital and NHK Technologies for the development of technologies and applications for immersive live experiences.

As a company at the forefront of technological innovation, we also have a wider goal to work together with partners across the industry to be part of the shaping of the future of immersive technologies and create new content and experiences.

Building on our previous EU funded project, immersify.eu, developed with our partners from Ars Electronica, PSNC, and with the collaboration of NHK Technologies, as well as our experience of live streaming the Tokyo 2020 Olympic Games in 8K, we are looking forward to developing new formats and experiences of the streaming performative arts. Ideas include presenting classical music and performative arts in immersive spaces, exploring the idea of augmented concerts and performances using interactive XR technologies, and finding ways of connecting remote audiences and performers. We imagine that such ideas can help to draw in new audiences, increase access and also be used in education.

To develop the concepts and technologies for meaningful, inspiring, interactive and participatory experiences of world class art in an intuitive and socially rich context, we are looking for an interdisciplinary network of partners in technology, science and arts. So this is also a shout-out for anyone interested to join us in developing the potential of these technologies!

Acknowledgments

• NHK Technologies, Japan. K. Wada and J. Terada.
• Spin Digital Asia Pacific, Japan. T. Kariya and A. Nagara.
• Sony, UK. Brian Edwards.
• Fraunhofer IIS, Germany. A. Murtaza.

Photos