Bad sound design is hurting your users: why audio matters for VR

Why is sound design so crucial for VR applications? What factors contribute to a good sound design? And how are those effects created? In this interview, Robert Vasenda (Head of Innovation Security, VR/AR and Mobile at MaibornWolff) discusses these questions with sound designer Mohamed Ben Jazia (Bnjmo, Senior Software Engineer Mobile & X-Reality at MaibornWolff), who has created several virtual experiences ranging from entertainment to virtual training or marketing showcases in VR.

Robert Vasenda: Hi Bnjmo and thank you for taking the time for this interview. Let’s start with a simple question: How would you describe what sound design is in the context of Virtual Reality?

Bnjmo: I would say that sound design is an important topic in almost every 3D application. Whenever you have active virtual objects, for example objects that you can interact with, our subconscious will expect them to be able to produce sounds. While photorealistic 3D rendering is very important to create immersive 3D environments, the focus on the auditive side is as much important to maintain the immersion.

VR is more specifically focused on immersion, so sound design is even more important there. When you are in a VR world and interact with it, you would expect to get auditive feedback on your actions. Say, you pick a cup up and then drop it, it falls, and you would expect to hear what happens. If there is no sound associated with it, that will directly break the immersion and pull the user from the virtual world.
 

Robert: So, would you agree that the sound design makes the immersion more complete in virtual reality?

Bnjmo: In my opinion, a good sound design sets the pillars for a good experience, but the user won’t specifically focus on it. I’d go as far as to say that good sound design is something you don’t necessarily notice. Bad sound design, however, will strike you right away and can totally break the immersion. So, the idea isn’t really to create the most complex or most magical sound you can, but to create something plausible and convincing enough to keep the user in the flow of the experience.

Robert: From what I understand now, good sound design is necessary for VR environments?

Bnjmo: Yes, I think that’s the most important thing about good sound design: It’s an integral part of the whole process of recreating the real world. We try to create a replica of our environment with VR applications – both in what we see and how we interact with it. And naturally, there is also sound involved in every interaction in the real world. It’s providing feedback, telling us if something is working the way we intended or not. If I put my phone on the table, for example, I expect it to make a particular sound, telling me it reached the table and that I can safely release it. If it doesn’t, then something is wrong, either with my mind and sanity or with the whole world.

I think we can learn several things from video games as well. In fact, they have been setting standards for this kind of applications for decades. There has always been a focus and importance given to the audio since the very first video games. Even with the 8-bit sound cards they had back then, sound designers always managed to create authentic and sometimes iconic sounds that contributed to the success of their titles.

These early examples and the evolution of sound design show how much importance the whole industry has been putting on audio in general. You can tell from the development of game engines and what they bring for the sound design side nowadays, from tools and technologies to advanced output formats.

Robert: When talking about game engines and sound, I think another relevant aspect is spatialization. What role does this play for sound design in VR environments?

Bnjmo: Spatialization or spatial sound might be one of the most important aspects of audio in VR. Spatialization is the ability to get 3D surrounding audio. Although spatialization is performed by the game engine and is not part of the sound design workflow for VR applications, I think it is still important to mention it as it reinforces the realism factor of the experience. A good spatialization should allow a VR user with closed eyes to determine where a sound is coming from. Therefore, it can even be used as a game mechanic to attract the user’s attention towards a specific point of interest in the experience. 

A concrete example is how we designed a VR app to simulate a potential feature for a smartphone app, used by car-owners. Its main goal is to help the users navigate to their parked cars. At some point, the user is in a garage, in a relatively dark area, making it hard for him to see. However, when he gets close enough to his car, it honks and flashes, helping him find it more easily. Thanks to spatial sounds, most users who tested the experience turned around in the right direction and were able to successfully locate the car afterwards.

Robert: You already touched on what is good sound design. For a user in a VR world, it’s sound that at least isn’t disturbing or even a sound design that strengthens a genuinely immersive experience. Is this the only important aspect of good sound design, or are there more contexts and factors? 

Bnjmo: Yes, there are two aspects to good sound design. The first one is what we already discussed: You try to recreate existing sounds and want them to be a good replica of what the sound is like in the real world. You look at what is happening in the 3D experience and recreate the right sounds.

The second aspect is more abstract, and you could say, more related to imagination. It’s about creating synthesized sounds and responses to interactions with virtual environments. For example, let’s say you are doing a virtual training and getting instructions what steps to do in a factory setting. When you do it right, there will be positive feedback, a kind of gratification by sound. And when you don’t, a sound should also give you feedback, telling you to approach the problem in a different way.

This second aspect is even more prominent than the immersive experience we talked about: Sound design is the active interaction element between a user and an application. The sound is used as a medium to communicate with the user. When designing the sound, in this case, you’ll think even more about the emotions and the mood that you want to convey. So, for positive feedback, you would be trying to create something more vibrant and happier, for example a crescendo kind of melody.

If you want to provide negative feedback, then you are aiming for sad emotions. In any case, it needs to be a sound that everyone understands right away. You cannot just put some random sound found on the internet and throw it into any interaction. That would make no sense to the user. It would be like composing a piece of music by randomly adding notes. You need a coherent composition combined with the right sound design to ultimately give a sense, a meaning to your sound.

Robert: So, if you need a specific sound for a design, just browsing through all those online libraries isn’t going to work?

Bnjmo: Exactly. Of course, it might be possible to find an excellent sound in a library, but then again, you will need to create a different atmosphere for every application and experience you are building. After all, you are not creating sound to isolated elements, but rather to elements that, as a whole, form a world. Naturally, this will differ between the environments and from aesthetic points of view.

The sound needs to be aligned with all these aspects to feel realistic. Let’s say you have a futuristic design; then you would go towards this specific direction in the sound design, possibly a little on the science fiction side. Or maybe your experience is rather simplistic, so you would pick a more lightweight sound design. You will always aim for harmony with the whole environment, including visuals and sound. And of course, sound design is a creative process, which means every sound designer will produce something different according to his taste and preferences.

Robert: Let’s get back to the immersive experience you talked about before. When you create a very real sound, I, as the user, will feel like, “wow, this is true, this is the real thing.” How does this work from a sound design perspective? Will you be using the real-real sound that you would have in reality, or do you create something that makes me have the impression this is real?

Bnjmo: It depends on the workflow of the sound designer. Some sound designers will go into the wild, take a microphone and try to record whatever sounds they find around them. Some try to completely recreate the sounds instead using all sorts of objects. I don’t know if you have seen them, but there are some great documentaries about how they recreate each sound for movies. In the acting scene, they only record the voices; all other sounds will be recreated afterwards in a dedicated studio, where they have stuff like blankets and rice that they move around or punch. Then, they mix these sounds to create effects like for flying birds or crushing bones.

Sound design for the type of 3D applications we discussed is not very far from other forms of sound design. I think we can borrow some workflows from both practical and digital approaches in cinema or video games. It’s just recording or synthetizing and processing the sound. I put it in software, mix it down, and maybe put some layers of sounds together to create a new one. That’s just a matter of creativity.

Robert: This sounds interesting. Can you tell me a bit more about the workflow?

Bnjmo:  Every sound designer works on a DAW, a Digital Audio Workstation, which serves as a basis software for making the sound design. A good analogy from programming would be an IDE, and just like it’s the case with IDEs, there are also different DAWs out there, like FL Studio, Ableton Live, Cubase, etc. They all have different interfaces and tools, but in the end, they are generally the place where you put your sound file in and modify the sounds by mixing them, applying equalizers, some effects, layering them, changing the pitch, and so on. It’s basically the workstation for every sound designer. 

From there on, the sky is the limit, and it’s all about the imagination. I’ve seen some examples of dragon sounds created from the horn of a car that was just pitched down and modified by effects. And in the end, it does sound like a real dragon, even though at the start it was just like a regular sound from a car, just a honk. It’s so fascinating how things can be transformed.

So far, we mostly discussed the practical approach; you start from a recording or a sample and modify it. But you can also synthesize your sound to create new sounds digitally. There are different synthesizers, which are plug-ins to add to your DAW. These will then give you powerful oscillators, so you can define or select a specific waveform and then synthesize it at defined frequencies, add some harmonies on top and modulate it or add effects to come up with the desired digital sound. This is mainly used for creating UI sounds or just generally more science fiction types of sounds that are more virtual and non-organic. The workflow for sound design would usually be a mix of both approaches. You use both practical and digital sounds to create amazing mixtures.

Robert: With all those tools and processes involved, what would you say, how much creativity and how much engineering is involved in sound design?

Bnjmo: I would say sound design takes the best of both worlds, so it is a very technical process and, at the same time, requires a lot of creativity. As I just described with the whole workflow, it can get deep in terms of technical challenges, so some background knowledge about sounds and digital sounds can be beneficial to understand better how the software processes the sound.

Then you would have the part of sound engineering in the sense that you need to mix and master the whole sound you’re working on, so it aligns with industry standards regarding normal volume levels and adjusted frequencies.  That is the engineering side. You need to understand these processes to get professional results, and there are many technical aspects that you need to master to produce sounds of good quality.

But then again, there isn’t just one specific solution for every given problem and not a pattern that you can follow just like that in every case. When working on a particular sound, you cannot just grab the perfect solution from the internet, but you must figure it out yourself and come up with your answer on how to do that. If ten sound designers would work on the same problem, none of them would create a sound exactly like the other. That’s why sound design is also a very creative process. So, to sum it up, as a sound designer, you need to have a good understanding of the sound engineering side and a creative mind.

Robert: Now, we have talked about sound, but what about the opposite – would you say there is a perfect silence?

Bnjmo: The perfect silence – I’m not sure I understood the question; what do you mean by that? 

Robert: I’m thinking of a situation where you hear nothing at all. Because, for instance, when I’m putting my earphones on, I will sometimes hear my heartbeat or steps on the floor. Or if I sit somewhere, there might be a car honking. I cannot recall if I have experienced perfect silence in, let’s say, the last five years.

Bnjmo: Fun fact – there is a place like that: The Anechoic Chamber at Microsoft Labs. It has acoustic phones and perfect sound isolation, so no sound from the exterior can penetrate it. It’s known that most people cannot spend more than 10 to 15 minutes in there because they will just start going insane. They will begin to hear sounds that are not there because their brain would try to composite for the lack of sound. Or they will even start hearing their heart pumping, which can be quite scary for some people because they are not used to that.

That is a perfect example of why we cannot create a VR application without sound. Nothing resembles that in the real world. Even when you’re alone at your home, some sounds ambient are happening. Maybe some footsteps from the neighbours or some furniture cracking. There are always tiny sounds that we are used to, and we’ve been living with our entire lives. Knowing that, building an application without any sound is just insane, so to speak.

---

About the authors