Unearthed - Interactive 360˚ sound and video in a web browser
Posted by Chris Pike on , last updated
This week we launched a new pilot on BBC Taster called Unearthed. It's an interactive and immersive story following a day in the life of a hummingbird, featuring binaural sound and 360˚ video. The project, run through Connected Studio, was a collaboration between the Natural History Unit, R&D's audio team and an agency called Realise.
We worked with sound designer Richard de Mowbray to create an immersive headphone mix for Unearthed, using our binaural production system. If you've not heard of binaural sound before, check out my previous post. In short, it is a way of recording or processing sound to create immersive 3D scenes specifically for headphone listening.
BBC R&D - Binaural Broadcasting
BBC R&D - Fright Night - Binaural Horror
Most of Unearthed uses normal 2D video, albeit with some nice interactive touches. For these scenes we created a binaural mix from stereo and mono recordings, much as we did for the Tommies radio drama production last year. We used various tools to construct a 3D soundscape from these conventional recordings, including 3D panning, upmixing, and 3D reverb. These tools are important when working on real productions, as you can't always capture the ideal source material for 3D audio; as much as I'd like to have taken a microphone rig like this to the jungle.
For us the most exciting part of this project is the 360˚ interactive video scene, because it also has 360˚ interactive audio! Meaning that when you rotate the scene to look around, the sound rotates with it. This is when binaural sound technology really works well, when you can interact with the scene. In our lab we've been using head tracking with our binaural production system to do this for some time. We showed in an experiment last year that this can lead to a highly plausible impression of sounds coming from outside of the headphones in the room around you.
For this reason, binaural sound will feature in virtual reality systems when they hit the market in the near future. We've already got some experience working with VR and immersive audio. We've been doing production experiments ever since we got our hands on an Oculus Developer Kit. Earlier this year I worked with Björk to create a VR music video for her song Stonemilker, which was exhibited at MoMA PS1 in Brooklyn. The video is now available on YouTube, although unfortunately YouTube doesn't yet support 360˚ sound. So that's why we're so excited about Unearthed, as far as we know, it's the first time that interactive binaural sound has been presented with 360˚ video in a web browser without needing to install a plug-in.
It's all made possible through an emerging technology called Web Audio, which allows developers to efficiently process audio signals in a web application. The Web Audio API is rapidly moving towards standardisation, my colleague Matt Paradis is co-chairing the working group in the W3C that is making this happen. Several major browsers already support the draft specification and many demos already exist. This year also saw the first Web Audio Conference held in Paris, at which our team presented several papers and demos.
BBC R&D - W3C Audio Working Group
To produce the sound for this 360˚ scene, we used a novel tool we've developed that combines our real-time binaural rendering software with 360˚ video rendering for the Oculus DK2 headset. The sound source positions are displayed as overlays on the video, which is presented in sync to the audio from our workstation as we're working. This allows the engineer to be immersed in the scene whilst mixing it and to interactively explore it as they work.
The next challenge is to export the scene so that it can be rendered by the web player. We've helped to standardise a meta-data model for representing advanced audio content called the Audio Definition Model. This can be used to translate the experience between different systems, such as our native production renderer and the web player. We also applied some pre-processing to reduce the complexity of the scene for the web player, this is one aspect that we plan to research further.
Web Audio is exciting because it makes experiences such as interactive binaural sound accessible to a large number of people without them needing specialist equipment or software. It does not yet offer the same sound quality and efficiency as can be achieved with a native application, but as the standard develops we believe the gap can be narrowed. It's important for the BBC that new content experiences are available to as many people as possible. We will continue to work on the production and client-side technology for binaural sound and 360˚ video, and hope to bring you more immersive and interactive content experiences soon.
I'd like to thank everyone at the NHU, Realise, Connected Studio and R&D who worked on this project. Please do try it and give us some feedback. I hope you enjoy it!
P.S. Check out this wonderful and somewhat related video on the BBC News website.
Tweet This - Share on Facebook
BBC R&D - All our Audio Research
BBC R&D - Audio Definition Model Software (Open Source)
BBC R&D - Binaural Broadcasting
BBC R&D - Virtual Reality Sound in the Turning Forest
BBC R&D - A Virtual Reality Fairy Tale Premiered at Tribecca Film Festival
More on Virtual Reality and 360 Video:
BBC Connected Studio - Watch 20 minute talks from our experts on Virtual Reality and 360 Video from #BBCVR day
BBC R&D - Factual Storytelling Tips for 360 Video
BBC R&D - Enhancing 360 Video with Graphics in the Large Hadron Collider
BBC R&D - Experimenting in 360 with The Kraken Wakes
BBC R&D - 360˚ Viewing Scenarios Study
BBC R&D - 8 Tips for Producing VR Projects
BBC News Labs - 5 Lessons in VR
BBC R&D - 360 Video & Virtual Reality
This post is part of the Immersive and Interactive Content section