Mixing Video and Audio
So recently I had a flash of an idea, opened up Chrome Canary and started messing around with the bleeding edge of HTML5.
The basic idea was to use audio to disrupt some video.
The louder the audio, the more disrupted the video becomes.
After a few hours, it looked like this.
And now, it looks a little something like this :
So, how does it work?
- Firstly, I use the getUserMedia API to grab a user’s webcam and microphone stream.
- This is then hooked up to the Web Audio API to analyse the microphone audio in real time.
- My audio analysis engine then dynamically updates a fragment shader*.
- And finally, I use WebGL and Three.js to apply the fragment shader to the user’s webcam feed.
It took a lot of tweaking to get the effect just right. Hopefully it works just as well in your browser as it does in mine… please let me know if you have any issues!
Just make sure you use a version of Chrome greater than v23.0.1271.101. And then, very importantly go to chrome://flags/, enable “Web Audio Input” (near the very bottom) and then relaunch Chrome. Now you’re ready to go!
This now works in the current version of Chrome, rather than just Chrome Canary.
However you still need to enable Web Audio Input as mentioned above.
* Shader inspiration and knowledge gained from the awesome Shader Toy.