Board index Technical Discussion General Technical Discussion Positional Audio

Positional Audio

Technical discussion forums for the Oculus Rift Community

Post Sat Apr 06, 2013 10:09 pm

Posts: 18
Location: Durham, NC

Maybe this belongs in the developers' forum, but using rift rotation data to do positional audio would be cool, and wouldn't take much CPU.

http://en.wikipedia.org/wiki/3D_audio_effect
http://www.youtube.com/watch?v=IUDTlvagjJA

Post Tue Apr 09, 2013 8:09 pm

Posts: 26
I'm trying to wrap my head around how this would work...so the game would have to take the source audio originally recorded in a studio, use OR data to position a virtual dummy head appropriately in the game, rerecord the audio on the fly to account for head shadow and then play it back to create the illusion of 3D audio? Or am I looking at this the wrong way..

IMO 3D audio is what game devs should be focusing on, not pushing pixels. I also think 3D audio is the key ingredient to make OR truly work in terms of immersion.

Post Tue Apr 09, 2013 8:18 pm

Posts: 3
I'm very interested in what people can do with audio and the Rift.

I dont think binaural audio is the way to go necessarily though. Any audio files you would want to use would have to be recorded with a dummy head. And in the case of the virtual barber, what happens when someone using the rift turns their head? The razor that was behind their head would now be right next to one of their ears but the audio would not reflect this. I'm also making this statement on the assumption that there is no DSP to process audio to make it sound like it was binaurally recorded. No one has developed that yet.

With that said, I think sound design for the Rift will be more challenging. Users will be much more aware of the environment that they are in, and it will be more obvious if something sounds off like in the case of improperly used reverb or lack there of.

Post Tue Apr 16, 2013 4:49 am

Posts: 2
Hi, I'm a audio dsp developer from Malm? Sweden. I soon have BsC in sound programming (acoustics, filtering etc)
I think the Rift could provide the incentive for devs to finally invest in proper "auralization", which is calculating all the psychoacoustically important "early reflections". Why? Well, with the rift, the discrepancy in quality in acoustics calculation will become very apparent just like a lot of focus needs to shift in what details are given more work in graphics.

There is a lot of previous interesting work that has been done by the GAMMA group at the University of North Carolina. Why these efficient and very realistic auralization algorithms haven't been adopted yet is a true mystery to me. Game audio hasn't developed much in ten years, except for a few exceptions (Frostbite Engine is MILES away from any middleware or other engine).

Game audio needs to go procedural like the rest of game design, but the reason why this is proving so hard is that most sound designers come from a music background and have no programming skills. Procedural Audio is very similar to shader programming, you need some math skills and especially acoustics is very boring with a lot of differential equations and transforms. I'm lucky to be one of a few that has received an education in procedural audio. The industry needs to evolve...

This is how an auralization system for the OR would work:

A HRTF model where the head and sholders are detached is precalculated. The head moves according to the OR sensors. The shoulders are oriented based around the player model (arms etc go into this equation). As sounds are generated in the enviroment, its direct source, multipath reflections and late reflections are sent to the precalculated model and filtered, this gives you a very realistic sense of being in the enviroment because you can tilt your head to help localize sounds, we do this subconsciously all the time and I think it will provide a much higher level of immersion. The shoulders are very important for localizing sound in the height axis as well as distance. The brain uses all of these "reflection cues" to create an inner image of the enviroment, a bit like bats.

Post Tue Apr 16, 2013 2:03 pm

Posts: 3
Yeah I certainly believe and hope the OR does more for sound. I do think that it will do more for spatialization via reverb and the like rather than HRTFs. I've never heard any HRTF that works for an arbitrary audio source. I've only heard things recorded via HRTF. Are there examples of converting a source to sound as if it was recorded via HRTF?

That GAMMA group does some awesome stuff. I would love to see an OR game with directionally-varying reverb.

Post Fri Apr 19, 2013 6:34 am

Posts: 2
All you need to do is render out first into a 3d multichannel arrangement of your choice, let's say it's an ambisonic format, then you need to convolve every channel with an FIR response that corresponds to that channel (these need to be prerendered for the HRTF), for every channel you get a stereo stream that you then mix and that gives you your binaural sound.

To get a really good model you could let people input their skull circumference at some point and also their height and render a better response, this is kind of important because the response is unique to every person.

Post Fri Apr 19, 2013 1:22 pm

Posts: 9
Most of you guys are overthinking this. You don't need to re-encode anything. In the game, all the sounds are mono sound sources placed about the player in 3d space, and then the game figures out the stereo representation based on the player's position. All that is needed is to improve the algorithm so that it takes the position of the head into account as well. What Veqtor described would be an ideal implementation, but we can begin by just placing the virtual microphones on a virtual head and linking it to the headtracking.

Post Sat Apr 20, 2013 7:49 pm

Posts: 18
Location: Durham, NC

I am by no means an expert on this, but I assumed that amplitude and phase shifts to each ear based on the pose (rotation and position) of the head relative to the virtual sound source would be good enough.

Post Sat Apr 20, 2013 10:31 pm

Posts: 9
That would be a start. At some point you would want the HRTF system that Veqtor described, but I think we can start with just changing the stereo microphone position based on head position. VR is a brave new world, we gotta start somewhere.

Post Mon Apr 22, 2013 6:38 pm

Posts: 2
DieKatzchen wrote:
Most of you guys are overthinking this. You don't need to re-encode anything. In the game, all the sounds are mono sound sources placed about the player in 3d space, and then the game figures out the stereo representation based on the player's position. All that is needed is to improve the algorithm so that it takes the position of the head into account as well. What Veqtor described would be an ideal implementation, but we can begin by just placing the virtual microphones on a virtual head and linking it to the headtracking.

Wouldn't it already be linked to headtracking? I mean, if you play Half Life 2 and you move your mouse around when someones talking, you'll hear it from your left side or your right side or wherever the source is relative to where you're looking. Isn't moving your headaround with OR the same as moving your mouse around? I haven't been able to test this out on the OR yet.

I think to get a quick and dirty immersion effect you'll need to have a virtual microphone for each ear (which will shift along a 2D plane) and then perfect the reverb of the 3D environments based on the players position in the room.

Next

Return to General Technical Discussion

cron
Spinnakr Active Analytics