• Announcements

    • Spaff

      These Forums are closing!   10/04/2019

      After more than a decade of serving this community well, these forums have finally run their course and it's time to close them down. That doesn't mean we want to close the doors on our community, quite the opposite!
      Our discord server grows ever busier by the day, and we encourage all Double Fine fans to meet us over there www.discord.gg/doublefine In a short time these forums will become a read only archive and will remain that way until they become needed again.
      You never know, it might happen.  There is... a prophecy. Thank you all for being part of these forums, and remember that the fun is definitely not over - so please join us on Discord! Love ya, Spaff, Tim, Info Cow, and all of Double Fine.
Sign in to follow this  
DF Anna

Programming Update #5: Audio Implementation

Recommended Posts

Hi Anna, (This is also my first post in the forum),

I'm really interested to hear more on this as I am an aspiring sound designer/composer and understand that games companies appreciate an understanding of the audio integration. While I've looked into FMOD before, I've lacked any sort of guidance to how best to make use of it, or how it really works at all - so this is very exciting for me and I can't wait to hear more from you!

-Will

Good to have you, Will :D! I'll post some more detailed stuff as soon as I get the chance that will hopefully be relevant to you.

Share this post


Link to post
Share on other sites

First up, this isn't a criticism of FMOD or the direction you lovely people have taken - just a question out of curiosity :)

Was the non-freeness of FMOD something that was taken into consideration when deciding on your audio implementation?

Share this post


Link to post
Share on other sites
Was the non-freeness of FMOD something that was taken into consideration when deciding on your audio implementation?

There simply is no free implementation of anything even remotely like FMOD Designer or Wwise. I've been looking very hard for one for a while now.

Share this post


Link to post
Share on other sites
Was the non-freeness of FMOD something that was taken into consideration when deciding on your audio implementation?

There simply is no free implementation of anything even remotely like FMOD Designer or Wwise. I've been looking very hard for one for a while now.

I said FMOD, rather than FMOD Designer. I'll agree that Free audio engines aren't that prolific or as well featured - I'm just keen to know if it was a factor in the decision making process (since it seems to have been a consideration when selecting Moai).

Share this post


Link to post
Share on other sites
Anna,

I wonder whether you have evaluated using Audiokinetic Wwise instead of FMOD Designer/FMOD Ex, and if so, what you think about it. I guess going with FMOD Designer is the better idea, since you already have the full pipeline and experience with it, but Wwise already implements a full dialog system.

I've looked into both, and I'm quite undecided on which I prefer more. FMOD Designer's editor is definitely much worse than Wwise's, though.

I'll answer for Anna as I helped w/ an eval of Wwise a couple times in the last few years.

We considered FMOD and Wwise early on, way back at the start of BL. At that point, Wwise was brand new and FMOD was just getting Designer off the ground. Wwise's tools were already very sexy, but the license was super expensive and their runtime much less proven/mature than FMOD. Once we picked FMOD for BL, we invested a lot in game integration, training, build integration, etc. We also built a really good relationship with the FMOD devs which was super key during a few critical milestones.

After BL we took one more look at Wwise, but by then FMOD Designer had matured a lot and closed the gap in terms of sexiness. We could do pretty much everything we wanted with FMOD Designer + custom stuff and the cost to switch in programming time and training would have been significant. It was a classic "if it's not broke..." situation. Also, while Wwise has become a lot more cost competitive, it's still not on par for most of the games we make.

Since then we've continued to benefit from their support, including in this decision to bring our FMOD Designer integration to Moai. We've also been involved in the design of FMOD Studio since way early, and we expect that once FMOD Studio is released (and the kinks are worked out :) that FMOD will have very comparable tools.

Regarding things like voice, most of our time has not been spent in the sound runtime part, but in the tool for authoring the dialog, localizing it, tracking changes, etc. That's custom with Wwise or FMOD, so it was not a big factor.

To be clear, I think Wwise is a really good product, too. They definitely set the bar for audio tools quality and pretty much forced everyone else to step up their game on that front. They've done some interesting realtime work, too, with granular synthesis and more advanced DPSs in particular. They've done all that while becoming a lot more cost competitive. If I were evaling FMOD vs Wwise w/ no existing investment, especially for a big game w/ a big audio team, it would be a very close thing indeed. But that's not the situation DF is in right now :)

Edit: Anna just reminded me that Wwise doesn't support Linux (yet) either. One more big reason for FMOD for Reds.

Finally, an audio post! I've been lurking until one was put up.

What's your general impression of FMOD Studio? I saw it at both GameSoundCon and GDC, both times running Stacking. It seems mighty impressive, especially the support for hardware controllers and real honest to god plugins. How soon after release of it do you think you guys will/can adopt it?

How big is the audio team for this game, and how big is it overall at DF? What's the ratio of in house vs. contractors?

Also, need an audio intern? Never hurts to ask ;).

Share this post


Link to post
Share on other sites
In other words, if you loaded all your sound into memory at create time, it doesn't matter for latency that you were using ADPCM.

This is a really, really important point. Colton asked about using uncompressed audio for action games, which implies that compression causes latency, which is a dangerous assumption because it's not accurate. In audio, the main cause of latency is file IO. If you need to hit the physical media to play a sound (because it's streaming or in memory but just hadn't been loaded yet), that'll be a penalty in the 10s or 100s of ms (esp if you are hitting slow optical media like a DVD drive). The second main cause of latency is buffering in the mix chain, and then in some cases buffering that's specific to the codec implementation. Even if a sound takes a lot of CPU to decode, it's unlikely to play late. If you don't have enough CPU to decode a sound it'll just playback stuttery, with gaps between audible bits.

Generally the main choice you make when choosing a codec is a CPU time vs memory tradeoff. The more CPU time you burn, the less memory you'll have to use. Fortunately, most game platforms have a single, custom format that they can decode in hardware, which makes it basically free to use (i.e. XMA on 360, VAG on PS2, MP3 on SPUs on PS3, GMADPCM on GC and Wii, etc). Other formats can be decoded, but only in software (i.e. on the CPU), which may be expensive depending on the codec and CPU type. Mobile devices are a bit odd in that they generally have hardware to decode a single "heavy" format like AAC or MP3, but then everything else has to be done in software.

Absent specific hardware, PCM and ADPCM/IMA4 (4:1 compression) are pretty close to free to decode, even on mobile phones. MP2, MP3, Ogg, CELT, WMA, etc. are all much heavier codecs. A modern PC can afford to decode multiple instances of pretty much any codec in software (provided the implementation is reasonable) but mobile devices tend to struggle with anything more than ADPCM (save the one hardware decoded stream).

And this is the kind of post I wanted to see! And thankfully I understand it, most gaming archives I've browsed through in the past, forms of ADPCM always occur, even right up to "big" name games like Grand Theft Auto IV on PC, I've looked through the radio files there and they all use ADPCM Single Channel Files, so things like songs on the radio, that's two mono files playing, one for left and one for right, and that's how the game mixes it on the fly when you leave a vehicle, so it would lower the volume, but depending on where you were standing near the car, it would raise one of the volumes so you'd hear more of say the left side of the vehicle (if you got out that side) or the Right side of the vehicle (if say you took cover there) and whilst impressive it may be, I always think about how much of a strain that must be to always have to keep these files in sync, even if it's just a vehicle driving past in the street, that NEED'S to be in sync right away if you decide to Steal that vehicle and the radio goes full blast. And that's partially why they opted for ADPCM here instead of using OGG like they did with the Radio Formats of San Andreas.

Share this post


Link to post
Share on other sites
Finally, an audio post! I've been lurking until one was put up.

What's your general impression of FMOD Studio? I saw it at both GameSoundCon and GDC, both times running Stacking. It seems mighty impressive, especially the support for hardware controllers and real honest to god plugins. How soon after release of it do you think you guys will/can adopt it?

How big is the audio team for this game, and how big is it overall at DF? What's the ratio of in house vs. contractors?

Also, need an audio intern? Never hurts to ask ;).

FMOD Studio is pretty awesome! We're definitely looking forward to the first release version. We're especially looking forward to the improvements they are making to multi-designer workflow. Right now we do that through some very careful splitting of projects, but it'll be awesome when we don't have to do that. Hardware controllers and other sexy UI bits should also be pretty awesome.

It's hard to say when we'll adopt Studio internally. Firelight still has to release 1.0, and very likely there will be a number of issues to work out in the first few versions. The engine API will also be different, so we'll have to redo parts of our integration, maybe substantially. We also aren't sure what the license terms will look like. All that's to say that there are just too many unknowns to say for sure.

Most likely, we'll adopt it when we start a new project that really wants to benefit from Studio and is willing to do the to integrate it. That's how most of our tech adoption happens here - as projects need it.

Share this post


Link to post
Share on other sites
(hopefully the one who did Dialogue Tree, he is the best)

That'd be Greg Rice!

Greg Rice is my favorite dialog tree eva!

Share this post


Link to post
Share on other sites
So, will this audio engine have the ability to follow the characters as they walk and talk? Or say, walk on different surfaces? Because I hate in games where there is such a wide stereo field, but the dialog and character sfx are always front and center. Like Half Life... Great game, bad sound implementation. It's cool that when an NPC is making noise, it positions, but when the main character makes nois, it's so monaural. Sounds weird to me at least.

Half-Life had one of the best sound systems for it's time. One example is just how easy it is to define the acoustics of a room to make it sound like a small metal chamber or a large concrete room.

Share this post


Link to post
Share on other sites
Haha, did Anna also draw all of the Redbots? I want to put little glasses and a tie on him and give him a stick pointer.

All the Redbot fanart is mine, yes. Tools used for rendering above masterpieces: crayon on backs of index cards. :D

You draw pretty well! (You've got the depth of field going, and the proportions are pretty good, and I liked the birds wings.)

Always interesting seeing someone who can both program well and also draw.

The more I do art, the more I'm seeing that there is quite a technical side to it (e.g. with painting, it's ability to see, among other things), which makes me realise that it probably isn't all that different to coding, which I'm sure has a creative, artistic aspect to it, too.

Share this post


Link to post
Share on other sites
Haha, did Anna also draw all of the Redbots? I want to put little glasses and a tie on him and give him a stick pointer.

All the Redbot fanart is mine, yes. Tools used for rendering above masterpieces: crayon on backs of index cards. :D

You draw pretty well! (You've got the depth of field going, and the proportions are pretty good, and I liked the birds wings.)

Always interesting seeing someone who can both program well and also draw.

The more I do art, the more I'm seeing that there is quite a technical side to it (e.g. with painting, it's ability to see, among other things), which makes me realise that it probably isn't all that different to coding, which I'm sure has a creative, artistic aspect to it, too.

haha you are way too kind!

I can't draw to save my life, but maybe hanging out with awesome artists is rubbing off on me a little.

I'm a huge fan of Ed Emberley, though, and have a bunch of his drawing books. Highly recommended. Especially this one.

Share this post


Link to post
Share on other sites

Hey, I want to start a bachelor-project using open frameworks which has FMOD EX support (no designer), can you tell me how mighty it is? Can i archive the same things as designer? And will help me understanding FMOD EX also when working with FMOD Designer, which i want to use for an upcoming project with a guy who uses unity?

Do you think it would be a good thing to dig deeper into that? Somehow it seems that there aren't that much sound designers/sound programmer (students) that also have a musical background around here in Germany.

Also made an internship at some German adventure company. But they don't think at sound the interactive way as you do, also when implementing external sounds from studios in there bigger games, which is some kind of shame. No one in there company is working on real sound programming/design. Could be budget problems or sheer ignorance. Praise on Tim that he knows how important sound is. Germany does do good adventures - yes - but some business men play with the passion of the ones who love the genre.

Keep up the good work!

Share this post


Link to post
Share on other sites
Hey, I want to start a bachelor-project using open frameworks which has FMOD EX support (no designer), can you tell me how mighty it is? Can i archive the same things as designer? And will help me understanding FMOD EX also when working with FMOD Designer, which i want to use for an upcoming project with a guy who uses unity?

FMOD Designer's runtime components sit on top of FMOD Ex. If Designer is the house, Ex is the foundation and the plumbing. Understanding Ex will definitely help you understand designer, as well as some of the lower level aspects of sound programming.

Whether that's worth your time really depends on what you want to do. If your goal is to make a game or sound demo partnering w/ a sound designer, then you probably want to skip straight to Designer since it'll be *much* easier on said partner. If you want to learn the gory details of audio programming, you might want to skip FMOD entirely and write it from scratch in C++, taking to Audio Units on iOS or DirectSound/WSAPI on PC or whatever. If you want something in the middle FMOD Ex is a good option.

Share this post


Link to post
Share on other sites

Speaking of Unity V3.x, it utalises fmod but uses it's own abstraction layer and only supports a subset of fmod's featurelist. So if you're going to work with fmod in Unity you better look at what of fmod is exposed and how it works by consulting their docs. The sfx side of Unity a.o. offers audio listeners, 2D and 3D sound sources (including the usual features like distance setups, doppler, panning, pitch, ...), playback of different file formats including tracker formats, a couple of DSP filters (such as echo, distortion, reverb, chorus, high/low-passes), the possibility to grab and alter the audio stream in realtime by writing your own filters. Filtering and low level access is restricted to the Pro version. Designer isn't officially supported at this point.

Share this post


Link to post
Share on other sites

Thanks Nathan for all your help :) I think i will take a deeper look into both ex and designer. i have some basic understanding of programming but creating audio engines from the scratch is to hard for me and i want to spend more time in making music and sounds.

taumel what about this http://www.squaretangle.com/FMODUnity.html ?

From what i read this should work, but haven't tested it yet.

Share this post


Link to post
Share on other sites

It depends on which version/platform you're going to target (from the description it's built for V2, desktop only). So far i only worked with Designer on projects apart from Unity. Whilst Designer is some sort of industry standard, you also might want to consider which audio features you really need for your project because a) you can setup a number of things inside the Unity editor already whilst getting similar benefits, like for instance a low memory footprint (one sound instance) b) relative easily code your own designer like behaviours for more simplistic designer like features like variance, granular effects, polyphony, randomisation of pitch, ... for more advanced stuff you need to dig deeper into the subjects audio and editor, use/buy/write a plugin again or wait for which audio improvements V4 might come up with.

Share this post


Link to post
Share on other sites

Great post, and I love seeing these technical details.

You mention that Ogg is a 'heavy' codec, but there are a bunch of games on PS3 that shipped with 320kbit/sec Ogg assets and use SPU decoders. Please don't use MP3 -- some of the biggest game disappointments I've had on PS3 have been with muddy MP3 audio assets in BioShock and Dead Space. Both games could have had amazing atmosphere with their music and environmental effects, but it sounded incredibly flat compared to Uncharted and even early PSN titles (like Siren: Blood Curse) and most newer PSN titles. (Fret Nice also was a major disappointment -- a music-oriented game with *awful* sounding assets.)

Poor codec choice also makes it hard to understand certain characters (especially female ones) in many TellTale games. I could barely understand the primary character in their Jurassic Park game, likely due to deficiencies in codec and bitrate. Other games end up mastering/EQing their assets to try and compensate, but then it just sounds weird and detached from the environment.

It would be really cool to have 24-bit/lossless audio for the music and reasonable-quality Ogg (or even AAC) audio for the voice assets. Just please no MP3 :)

PS: You mentioned 7.1, and using a 24-bit surround-sound reverb/echo would be AMAZING. Especially in The Cave ;)

Share this post


Link to post
Share on other sites
Great post, and I love seeing these technical details.

You mention that Ogg is a 'heavy' codec, but there are a bunch of games on PS3 that shipped with 320kbit/sec Ogg assets and use SPU decoders. Please don't use MP3 -- some of the biggest game disappointments I've had on PS3 have been with muddy MP3 audio assets in BioShock and Dead Space. Both games could have had amazing atmosphere with their music and environmental effects, but it sounded incredibly flat compared to Uncharted and even early PSN titles (like Siren: Blood Curse) and most newer PSN titles. (Fret Nice also was a major disappointment -- a music-oriented game with *awful* sounding assets.)

Poor codec choice also makes it hard to understand certain characters (especially female ones) in many TellTale games. I could barely understand the primary character in their Jurassic Park game, likely due to deficiencies in codec and bitrate. Other games end up mastering/EQing their assets to try and compensate, but then it just sounds weird and detached from the environment.

It would be really cool to have 24-bit/lossless audio for the music and reasonable-quality Ogg (or even AAC) audio for the voice assets. Just please no MP3 :)

Ogg and MP3 have very similar quality/size ratios. If one game sounds bad with MP3 and another good w/ Ogg, it's much more likely because the Ogg file was encoded at a much higher bitrate rather than intrinsic differences in how they represent compressed audio. As for SPU decoding, that's pretty much a requirement for any non PCM/ADPCM sound on PS3, which is why Sony provides stock SPU decode algo's for several formats, including MP3. Most PS3 games are very pressed for main memory (the default place to put sound assets) and not all audio systems support loading audio into GPU local memory on PS3 so the heay compression is more likely a symptom of trying to cram a 360 game into a PS3 and relying heavily on reducing the bitrate to make it all fit.

As for DFA, well, that'll all be up to the team, targets for download size, etc. I'm sure it'll sound awesome on all platforms, though PC most of all.

PS: You mentioned 7.1, and using a 24-bit surround-sound reverb/echo would be AMAZING. Especially in The Cave ;)

If you play The Cave on PS3, you'll find that it's mixed natively in 7.1 (as are all of our PS3 titles post BL). The source assets are 16 bit, but the mixer is 32 bit internally and the output is 24 or 16 depending on what surround mode your system is config'd for.

Share this post


Link to post
Share on other sites

Going back to the first Sidequest, Tim said that they still record lines for Double Fine games at Skywalker Ranch. Is that going to happen for this game?

Share this post


Link to post
Share on other sites

Yay audio code post!

Cool to learn that you guys are using FMOD.

We're using Wwise these days, but the Firelight guys are good people.

Share this post


Link to post
Share on other sites

The more MB I can download, the better! Make it 10.000 mb with awesome sound, graphics and music!! Hyyyyyyhhhhaaaaaaa

Share this post


Link to post
Share on other sites

You mention that Ogg is a 'heavy' codec, but there are a bunch of games on PS3 that shipped with 320kbit/sec Ogg assets and use SPU decoders. Please don't use MP3 -- some of the biggest game disappointments I've had on PS3 have been with muddy MP3 audio assets in BioShock and Dead Space. Both games could have had amazing atmosphere with their music and environmental effects, but it sounded incredibly flat compared to Uncharted and even early PSN titles (like Siren: Blood Curse) and most newer PSN titles. (Fret Nice also was a major disappointment -- a music-oriented game with *awful* sounding assets.)

Poor codec choice also makes it hard to understand certain characters (especially female ones) in many TellTale games. I could barely understand the primary character in their Jurassic Park game, likely due to deficiencies in codec and bitrate. Other games end up mastering/EQing their assets to try and compensate, but then it just sounds weird and detached from the environment.

It would be really cool to have 24-bit/lossless audio for the music and reasonable-quality Ogg (or even AAC) audio for the voice assets. Just please no MP3 :)

Ogg and MP3 have very similar quality/size ratios. If one game sounds bad with MP3 and another good w/ Ogg, it's much more likely because the Ogg file was encoded at a much higher bitrate rather than intrinsic differences in how they represent compressed audio. As for SPU decoding, that's pretty much a requirement for any non PCM/ADPCM sound on PS3, which is why Sony provides stock SPU decode algo's for several formats, including MP3. Most PS3 games are very pressed for main memory (the default place to put sound assets) and not all audio systems support loading audio into GPU local memory on PS3 so the heay compression is more likely a symptom of trying to cram a 360 game into a PS3 and relying heavily on reducing the bitrate to make it all fit.

I really disagree that Ogg and MP3 are comparable at the same bitrates, especially for the majority of female voices, woodwings, crash cymbals, etc, etc. I do agree that MP3 can sound reasonable at 320kb/sec (as can Dolby Digital), but it won't hold a candle to the aforementioned games.

Does the audio system chosen for DFA (or The Cave) allow for loading audio into the GPU's local memory on PS3?

As for DFA, well, that'll all be up to the team, targets for download size, etc. I'm sure it'll sound awesome on all platforms, though PC most of all.

That would be incredibly disappointing, as I don't play PC games on my home theater sound system.

PS: You mentioned 7.1, and using a 24-bit surround-sound reverb/echo would be AMAZING. Especially in The Cave ;)

If you play The Cave on PS3, you'll find that it's mixed natively in 7.1 (as are all of our PS3 titles post BL). The source assets are 16 bit, but the mixer is 32 bit internally and the output is 24 or 16 depending on what surround mode your system is config'd for.

What about the DSP effects? are they also 24/32-bit?

Share this post


Link to post
Share on other sites

I really disagree that Ogg and MP3 are comparable at the same bitrates, especially for the majority of female voices, woodwings, crash cymbals, etc, etc. I do agree that MP3 can sound reasonable at 320kb/sec (as can Dolby Digital), but it won't hold a candle to the aforementioned games.

Does the audio system chosen for DFA (or The Cave) allow for loading audio into the GPU's local memory on PS3?

I won't debate you on the finer points of codec quality. It sounds like you have a very sensitive, trained ear. For folks like you, I suspect the differences are as pronounced as you say, though I would posit that most folks are probably not like you :)

Our FMOD integration does supoort loading audio into RSX local memory. We use it for voice and SFX like boss attacks, mission scripted sounds, etc. DFA uses this same implementation, but as a reminder, DFA is committed to PC, Mac, Linux, iOS, and Android.

That would be incredibly disappointing, as I don't play PC games on my home theater sound system.

When I saw PC, I really mean our higher end targets, which are PC/Mac/Linux computers plus possibly some higher end tablets.

What about the DSP effects? are they also 24/32-bit?

DSPs are part of the mix chain (or you can think of mixing as a final DSP), so yeah, 32 bit, though the low level details vary by platform.

Share this post


Link to post
Share on other sites

It's great to read you are releasing the sound engine as open source. That's just awesome :)

Share this post


Link to post
Share on other sites

So awesome to see an audio tech discussion here. Did not expect it at all. I'm among fellow geeks!

I like that you guys wrote your own layer between FMOD Designer and the lower level audio stuff (XAudio, Multistream, etc.). We did something similar on a smaller project for similar reasons I suspect.

Very excited about you open sourcing your audio stuff through MOAI! Looking forward to more updates like this!

Share this post


Link to post
Share on other sites

being an audiophile I like the special attention to every detail and in this case sound it's a hit point!

Great stuff Anna, can't wait to start hearing the first tests of sound : )

V.

Share this post


Link to post
Share on other sites
Sign in to follow this