Music with the Web Audio API

Rich Williams speaking at JS Monthly London in November, 2016
795Views
 
Great talks, fired to your inbox 👌
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

We’ll look into the building blocks of audio and how to connect modules together, use some music theory and maths to create a nice tune and some cool effects.


Transcript


My name is Rich Williams. I'm a front end developer and I'm going to talk to you a little bit about some of the stuff you can do with Web Audio if you're trying to create some music with it. This might be quite ambitious, I'm not sure I'll be able to fit any of this in, but we'll see how we go. I'll just go through a quick introduction of the Web Audio API, so the basics of it, and an explanation of the audio modules, which are the constituent parts of...that's sort of how you create the sound in Web Audio. I'll do some basic audio visualization and then I'm going to try and live code some stuff, which could go horrendously wrong, but we'll see how it goes. And then...we'll try and do that. And then I'll show you some examples if everything goes pear-shaped. So just a quick introduction to Web Audio. Before Web Audio API you basically had to use Flash to...or QuickTime or something like that to create any sound in-browser, which was pretty horrible. And since then we've got the Audio HTML5 element, which is useful but it doesn't really allow you to do anything complex. This is where the Web Audio API comes in. And it allows you to do some quite complex stuff. One of the best things about Web Audio API is that it's cheap and low latency, so you can do...it has an incredibly high rhythmic precision, so you can do drum machines, and music programs, and things like that. You can do...you can achieve quite a lot of cool stuff. You can apply effects and filters to sound sources, to generated sound sources, and to sound sources, like a sound file or something like that. And it's a relatively simple API, and only basic knowledge of audio is really required. More knowledge is much useful...very useful as well, obviously. Quick explanation of audio nodes. So these are the main ones we're going to be using. Oscillating node is...it generates a waveform, by default a sine wave, which is just a smooth soundwave. It's 440Hz, which is an A on a piano, just like the middle note on a piano basically. It has other waveforms, it has...you can have triangle, sawtooth, square, and these all create different sounds, different timbres. Another one we use all the time is the gain node, and that's basically a volume control. We use loads of these as well. Convolver node is used a lot. This creates the reverb effects, which is the ambient sound that you hear in music and sound. You can...with a convolver node you can take an audio source and you can take the audio profile of it, so you can actually give it any sound and it will work out the audio profile and then apply that ambient noise to that sound onto any sound you give it. And then the destination node is the output. There's loads of other audio nodes as well. We might use a few of these, audio compressor. But yes, we can do...it's quite diverse. This is a really basic setup. You've got an oscillator node connected to convolver node, that gives it the reverb, and then that's attached to a gain node which will reduce the volume of it slightly. And then that passes through to the output. Because it's [inaudible], it's quite reverby. So a slightly more complex setup. The good thing about this is...with oscillator nodes you have...every node has audio parameters connected to it. With an oscillator you have...frequency is one audio parameter and you can connect another oscillator to that frequency. And you could also do that to...like, an oscillator to the gain as well, anything like that. And you can basically create...in this instance we're attaching an oscillator to the frequency audio parameter, and that oscillator will push the frequency up and down and basically create a vibrato effect, if you know what that is, which is...it's basically a wobbly sound, the sort of thing that opera singers do. And then that connects to more gain nodes and through to the destination. You get that sort of wobble. Some really basic audio visualization just to give you an understanding of what the waves look like, the waveforms look like. Sine wave. Sorry if everyone's going deaf. It really messes with your ears, actually, after a while. This is...you can get an idea of what the...the shape of the waveform, and the shape of the different waveforms. So that's your triangle. There we go. These are the basic four that you can use. You can actually, if you're a bit of a wizard at maths, you can create a formula and stuff and create your own waveforms to do whatever...to create whatever sort of sounds you want. So this is... I was bored, it's just... Absolutely nailed it. I'm going to attempt some live coding, and if it all goes wrong I've already got the examples ready anyway, so it's all right. - [Man] We believe in you. - I don't, I don't at all. So to start with, I'm going to try and create a chord from this. So I'm going to start with just an A note in this array, and... Once...so what I want to do... so once you press play/pause you want to create an oscillator for every note in that array. I think this could all go pear-shaped. So we're going to create an oscillator for every note in the array, and then we'll set the frequency to that note and we will give it type. So I've got an empty oscillator array, so what I'm going to do is actually...actually I'm going to map these two. Right. So this will create a square waveform, square oscillator with that frequency of that note. And then once you create that you need to connect it to the... I just skipped out loads of stuff. You need an audio context to start with. - [Man] I can't believe you forgot that. - It's pretty useful, pretty fundamental. All right, so we've got the audio context. I'm going to create a gain node as well. The oscillator will connect to the gain node, the gain node will connect to the destination, and it will hopefully play. The gain, the value, [inaudible]. So the oscillator will connect to the gain node, and hopefully...let me just start it. And... so when this is off I guess we do... Right. So let's see if it works. Nothing! - [Man] [Inaudible] Did you refresh? - I thought I did. You're right, I didn't refresh. Fantastic. That's basically playing one note. Trying to think of other things we could try on this. Compressor... So I could put a compressor on this as well. Actually, I've created... I'll show you what I was doing earlier with the detuner. I'll create a detune oscillator. And for that I will need a gain as well to boost it a bit. If you want to change...pardon me, sorry. If you want to change, detune a frequency of another oscillator, the frequency has to be very low because you want it to be audible, so you really want to...you want to do every second or something. Once every second or twice every second would be 2Hz, so... actually, I want to do a bit more, I'll do it at 6Hz and then connect these together. So connect the detuneGain and then...so we're on creation of these oscillators and connect to that one, and then start this jobbie. - [Man] So you can stack it? - No, it's all right, it can sit there...it's fine. I broke it! I'm just going to [inaudible]. Sorry. [inaudible] Okay, right. I attached it to the oscillator instead of the oscillator frequency audio parameter. So that's the one. And it still don't work. Detune the gain. Oh, I didn't refresh it. There we go. All right, fantastic. So it sounds all <i>Doctor Who</i>-like again. Fantastic. All right, okay, so... I've got a few songs. So that does it for every note in the array. If you [inaudible] frequencies... it's got some weird notes actually, one of them is a bit off. I'm just going to copy this in, just in case I didn't do the same. So that is a chord, to start with. Creating a melody is a little bit more complicated, because we need...there's a time dimension. So I set this up a little bit, anyway. It's basically got all the stuff that the other example has but it's got...so we added a compressor to attenuate the sound a little bit and it's got some reverb as well. This is the...this is...it's sort of adjacent...this is the object format that I created. When you create a note you want to want to know the note, you want to know the start time and the duration of that note. And I've created note objects so I can refer to that and get the frequencies, because I don't know all the frequencies off the top of my head. With this... so first of all we need to create a play loop. This play loop will be run through...this will use request animation frame. Request animation frame...you want this to loop round, and when you hit play you want it to start a play loop. And when you press stop, use the cancelAnimationFrame. We're going to play...we'll start playing the loop. In this loop we want to loop through these notes. And we actually want...we want play flags for all those notes, because once the notes play we need them to be...we have a flag that says they've already been played so we don't play them over and over again, obviously. And when you press play, notesToPlay. So we'll add a played flag, false. We've got a list...an array of notes here, which will have all the notes we want to play. We'll loop through this and then we need to create a loop here to loop through all these notes and play them. So this needs to check if it's been played yet. Actually, wait a minute, it needs to check what time to play them. So it needs to know the current time. And the way you find out the current time is to use the audio context currentTime, which is sensible. And because this is running as soon as I load this it needs to have a start time. That is massive. So that sets the start time. So it loops through and get the current time that way. To check...so what we got to do, we want to check if the current time is after, is equal or just after when the note should start playing and then when it's that, so it's about as accurate as it can be, and then it will play the note and then it will set the play flag to true. You want to check if it plays...if it's not played and the current times. And then... Do note.startTime. So startTime, startTime. Okay, so then we want to do...oh wait, it's note.played. Note.played equals true. Now we can create our oscillator again here, another oscillator. So many oscillators. Type equals...what do you want? - [Man] Triangle. - Triangle. That's a really rubbish one. Oscillator frequency, note...we need the notes, note.note. This will connect to...the compressor is the first module I've got in my linked module. I'll connect that to that, and I've got a detune oscillator as well, a detune gain. So, I'll do that, connect that to the oscillator frequency. And then we want to start this and then we want to stop it at the current time, the ac.current that says stop takes a precise time according to the audio context. So we'll take the current time plus the duration of the note. And I think... I think that should work, but if it doesn't... It doesn't. No, it does. It was delayed by a second because I actually put one, but I can't put one through the first beat. I'm just going to add... Where are you? Okay, currentTime plus one. Just to see if this works. Any note suggestions? No? - [Man] D6. - D6, okay. Let's see what it sounds like. That sounds horrendous. - [Man] Maybe you should change the triangle. - Change the triangle? - [Man] [inaudible] - Oh, change from triangle? Yeah. I agree. - [Man] If you recreate it and you [inaudible]. - All right, we'll see in a bit. - [Man] I like room challenges. - Actually, one more thing is to use a tempo. I've got a tempo at 120, so using the tempo. Tempo is the beats per minute, so if you have 120 beats per minute you are dividing 60 seconds by 120 so you're having a beat every half a second. I want to include that. Set the current time to... and then divide that by... So that should... No. Okay, so I've already created the beats here. Okay, so those are the notes. Now we can combine these two, hopefully. There's something wrong with that. Maybe this one...same player. Okay, cool. If there are any <i>Final Fantasy</i> fans, you'll be going... So there we go, that's a little tune. Thank you, thank you. I've got an invisible link for some reason. Anyway, so I've created a few bits and pieces with this. Does anyone here have perfect pitch? No? Okay. It means you can hear a note and know exactly where it is on the piano. We'll skip that. I was just wondering if anyone actually...or you can give it a go. What do you reckon, what do you reckon this note is? - [Man] Quite high. - Quite high. - [Man] G sharp. - Did someone say G sharp? - [Man] Yeah. - G sharp. You're wrong, it's G. It was quite close, actually. Okay, that's that. As we were saying before, with reverb you can take any sound source, take a profile of that sound source and you can create some quite, like, massive sort of... Some quite cool audio sounds from other audio things. I was just...I created this weird automated thing ages ago and I just attached to...attached an audio to it and it's quite snazzy. You can apply it to anything visual or any sort of automated visual thing by just generating oscillators and then attaching some sort of scale to it. - [Man] [inaudible] - It's heavy. I thought about doing it. I have looked into doing drum beats and stuff. You can...I'll show you later, actually. There's a...that's pretty much it. That last link is really annoying, because that's... I still want to find it. Where it be? What's going on? All right, this one. So I created this little thing with... where you just sort of write in... so you give it...it's basically a lot of numbers but I've sort of... The good thing about CodePen is it reloads every five seconds. So with this I created sort of a scale. You pass it in a scale and you pass in a lot of numbers. These are numbers of the scale and the length of the note, and you can combine them into tunes if you're patient enough to write it all down. I did it at work when no one was looking. So that's pretty much it. These are a few cool links to videos but I won't show you all the videos, obviously. But there are some really cool things. Sam Bellen managed to combine...he basically created a guitar foot pedal with Web Audio, which is pretty cool. Paul Adenot does an acid bass track that creates all the elements with oscillators, so everything is created by the computer, by the browser except for the clack, I think, or something like that. But other than that it's amazing what you can do with it. And Mattieu Henri is more of an audio visual guy, someone sent me a video, and he does some really cool stuff. So yes. My thanks. ♪ [music] ♪