Mixing Video and Audio

So recently I had a flash of an idea, opened up Chrome Canary and started messing around with the bleeding edge of HTML5.

The basic idea was to use audio to disrupt some video.
The louder the audio, the more disrupted the video becomes.

After a few hours, it looked like this.
And now, it looks a little something like this :


So, how does it work?

  1. Firstly, I use the getUserMedia API to grab a user’s webcam and microphone stream.
  2. This is then hooked up to the Web Audio API to analyse the microphone audio in real time.
  3. My audio analysis engine then dynamically updates a fragment shader*.
  4. And finally, I use WebGL and Three.js to apply the fragment shader to the user’s webcam feed.

It took a lot of tweaking to get the effect just right. Hopefully it works just as well in your browser as it does in mine… please let me know if you have any issues!

Just make sure you use a version of Chrome greater than v23.0.1271.101. And then, very importantly go to chrome://flags/, enable “Web Audio Input” (near the very bottom) and then relaunch Chrome. Now you’re ready to go!

Launch the experiment.

**UPDATE**
This now works in the current version of Chrome, rather than just Chrome Canary.
However you still need to enable Web Audio Input as mentioned above.

* Shader inspiration and knowledge gained from the awesome Shader Toy.

Comments

  1. mattbilson
      |  
    October 10, 2012 at 5:07 am

    Great stuff, Azik!

  2. Wigglesworth
      |  
    October 12, 2012 at 4:22 pm

    Nice Matt!
    (spotted the Roast Beef tee as well)

  3.   |  
    November 8, 2012 at 7:56 pm

    nice t shirt billy!

  4. Marcel
      |  
    May 23, 2013 at 4:16 pm

    cool! where can I find the code?

  5.   |  
    September 19, 2014 at 12:38 pm

    I read a lot of interesting posts here. Probably you spend a lot of time writing, i know how to save you a lot of work, there is an online tool that
    creates readable, google friendly articles in minutes, just
    type in google – laranitas free content source

Leave a Reply

Your email address will not be published. Required fields are marked *