For some time now I’ve been working on a software sound synthesizer for JavaScript. It all began when I found out about the WebGL 64k competion. Obviously, creating a full featured demo in a 64k JavaScript program poses several problems. The two problems that I found most interesting were:
- How do we do music in a 64k JavaScript program?
- How well can you compress a JavaScript program?
In the “native” demo scene (e.g. compiled Windows programs), these are well known problems with many interesting (to say the least) solutions, but for JavaScript the situation is a bit different.
The most obvious different (except that the JavaScript demo scene is very immature in comparison the the rest of the demo scene) is that a JavaScript program is delivered in source form – not in compiled machine code form. This means, for instance, that a 64k JavaScript program can hardly fit the same amount of information and logic as a 64k native program.
For the music I decided to look at the 4k demo scene, and I found this very neat software synth called Sonant, which includes both a simple music authoring tool and the C source code for the player routine. Great! 🙂 This was the starting point for my JavaScript synth!
Now, JavaScript syntax is not all that different from C syntax, but there are some pitfalls when porting from C to JavaScript. Perhaps the thing that hit me the most when porting was variable scoping: in JavaScript the scope of a variable is function wide (e.g. you can not limit the scope of a variable by using brackets). Since the original code reused variable names across the main loop, I had some funny errors before finally getting it right.
Another tedious task was converting the exported music data (in a .h file) to JavaScript. I did that by hand, but if I were to do it again, I’d make a tool for doing it.
Then the obvious question: How do you play back the generated waveform in a browser? A few options exist:
- Use the Mozilla Audio Data API.
- Use the Web Audio API.
- Use the HTML 5 audio element with a fat data URI.
- Support all of the above.
Since option 3 is the only API that’s currently widely supported (and option 4 would make the program larger and more complex) I decided to go with the data URI solution. It basically goes like this:
- Create the raw wave data in some JavaScript array (this is what the synth does).
- Convert the wave data to a string-of-bytes in proper RIFF WAVE format (i.e. prepend a header and ensure little endian byte ordering).
- Convert the WAVE formatted string to base64 encoding using the btoa() function.
- Prepend a data URI header: “data:audio/wav;base64,”
And yes, it works!
What I found out, though, was that the synth ate huge amounts of memory (between 1.5 and 4 GB, depending on browser!). The main reason was that I was using regular ECMAScript arrays for the wave data. There are some differences in how browsers deal with such arrays, but they all have in common that they use more memory per element than the size of the actual data (in ECMAScript, every array element is essentially an object). Add to that the fact that numbers in ECMAScript are treated as doubles (i.e. 8 bytes per number)…
The solution to the problem spells typed arrays. Unfortunately, typed arrays are not that common (yet). The current driving force for typed arrays seems to be WebGL, which is just starting to emerge. But there is one kind of typed array available in browsers that support the canvas element: the CanvasPixelArray.
It may seem like an awkward thing to do, but by using the createImageData(w,h) method (on a 2D canvas context), you get a byte array with w*h*4 elements. This byte array is packed (N array elements cost N bytes of RAM). There were two issues, though:
- I wanted 16-bit sound data, so I had to do manual packing and unpacking of the elements (store one sound sample in two array elements). This required some more code, but on the whole turned out to be faster than regular arrays anyway.
- Some browsers (hrrr-o-mmm-e) don’t support image sizes exceeding 32767 x 32767, meaning that I couldn’t request an N x 1 buffer. The straight forward solution was to use a ceil(sqrt(N)) x ceil(sqrt(N)) buffer. Since the array has linear access, this didn’t introduce any more complexity (just some wasted memory).
Since I’m interested in performance I continued with optimizations, mostly consisting of minor tweaks (storing object properties in local variables, inlining simple functions, etc), and I must say that I’m impressed with the performance of modern JavaScript engines! Actually, compared to the pure C version (compiled with gcc -O3), the JavaScript version is more or less on par in Opera 11 now!
You can try out the result for yourself here.
Next up: JavaScript compression! …another time 😉
Incredible this thingy is less than 4K. Very well done! I am really impressed! 🙂
MP3 version – http://min.us/mvjoBkQ
had fun with deobfuscation and saving base64 audio data to disk (via PhantomJS and some Ruby code). thanks Marcus and hope to see more articles/source code on synth
Hehe 🙂 Nice job!
This is fantastic work. Would you consider letting others use your ported synthesizer? There does not seem to be anything else out there of this quality yet (especially not with the base64 export support for maximum browser compatibility). Most of the other libs I looked at use the Audio Data API and have serious problems with time sync, popping, etc. I have a project that could really use this, and I would be glad to give you credit if you favor a Creative Commons Attribution license over straight open source.
It’s on my TODO list. I got the approval from Jake (of Youth Uprising) to release the source code, so it’s in the pipe… There are still some things left to do to make it really useful though (such as a binary->JavaScript song converter, so you don’t have to do it by hand).
Awesome, thanks! If I can help in any way, let me know. My javascript-fu is quite strong, although I know basically nothing about sound synth (hence the reason I’m looking for a good synth instead of writing one).
If you can do FFT in Javascript, here are some sugestions:
1) You can implement PADsynth algorithm http://zynaddsubfx.sourceforge.net/doc/PADsynth/PADsynth.htm , which is very simple and it gives you very good sounds
2) You can implement Paulstretch extreme audio stretcher from here http://hypermammut.sourceforge.net/paulstretch/ (or you can see the minimal python implementation here https://github.com/paulnasca/paulstretch_python )
Paul
Interesting stuff. Synthesizing in the frequency domain is a really cool technique (you can do almost anything). However, this little demo is more about doing a very minimal synth (in only a couple of hundred lines of code), and I fear that an FFT implementation might be on the heavy side for that. For another project, it might be a good starting point though (I can imagine doing a synth almost completely in the frequency domain).
could we use this technique to create a audio player that protects its source from being simply downloaded ?
Well, you should be able to add some level of “protection”, e.g by using some simple form of encryption/scrambling of e.g and OGG file, send it via XHR to the client, which de-scrambles it and turns it into base64-encoded data URI that is fed to an audio element. It’s not very secure, and it would eat tons of RAM and prevent any form of streaming, but at least you don’t get a pure URL to download. 😉
Amazing stuff! I have been searching for something like this for ages and this is so beautiful. Please release the source code, I’d be really happy to see it.
On the issue of memory and speed on the buffer array. Couldn’t you just use a raw binary data variable and write to it with bitwise shifting? This way you would not need to format your data array for each buffer chunk.
I’d love to see this implemented with more realtime interaction, like a keyboard. But it’s a good song, thx for making this. You rock!
Thanks 🙂 Actually, search around this forum and you’ll find some more updates on the subject. For instance, see this post.
Holy sh17 that’s nice! I’m a big tracker fan.
For something so hacky (by necessity, since js doesn’t have performant arrays yet) your code is really beautiful. Seems so simple once you explain it, but I know I’d be lost if I sat down to port the C code like you did without the benefit of a road map. That’s the definition of good work 😉
To clarify: “good work” is that which makes simple what would seem complicated to *others in general* – not just to *me* 😉
You might want to use Blobs and Object URLs for saving in browsers that support this. I suspect you’ll get less problems with this than with data URLs (URL length/memory). Even if you use data URLs you don’t have to use btoa (it isn’t in any standard). You can use: window.location = “data:application/octet-stream,”+escape(data);
(If data represents text and contains characters > 127 and shall be encoded as UTF-8 use encodeURIComonent instead of escape.)
Here I wrote about several methods to save JavaScript generated data:
http://hackworthy.blogspot.co.at/2012/05/savedownload-data-generated-in.html
Simplified it goes like this:
var blob, url, builder = new BlobBuilder();
builder.append(data);
blob = builder.getBlob(“application/octet-stream”);
url = URL.createObjectURL(blob);
window.open(url, ‘_blank’, ”);
URL.revokeObjectURL(url);
If you like you can do it a bit nicer for Chrome >=14 using an anchor element with an download=”filename” attribute and simulating a click on it (src is the same object URL). This will start a download no matter what mime type the blob has (you don’t need to use “application/octet-stream”) and it won’t flash a new window (what window.open does) and there is no chance of replacing the current document (what window.location = … might do).