Computer Vision

I'm fascinated by how a machine sees the world. It's capability for doing things like perceiving detail and detecting patterns far surpasses ours, but its ability to infer context and meaning is only as good as the education we give it. As a result, what it dreams up runs the gamut from bungled to brilliant, but unquestionably offers enormous potential to aid in our own ability to interpret and more deeply explore what's around us.

Face Substitution

Mapping significant facial features between the camera feed and features detected in the photograph.

Face Smashing

Distorting the parameters of the feature mappings

Head Tracking as a Controller

We rely so heavily on our hands for controlling games, I think we often forget there are often more intuitive interfaces at our disposal. Here I'm using the Kinect to control a character in Unity. His head movements mimick mine, and when I squint, the camera zooms in.

Controlling Audio with Facial Tracking

Another intuitive use of the facial tracking I can imagine would be to drive audio. Our mouths take very specific shapes to produce specific sounds and accents. I can imagine an application where facial tracking is used to give voice to animated characters, or as I'm experimenting with this video, mapping different features to different audio characteristics, such as pitch / frequency.

Controlling Music with Skeleton Tracking

Music is so tied to body movement, why not let your movements control the music? Imagine a DJ driving sound not through a giant MIDI controller, but through the movements of their body. While my dance moves don't exactly measure up, here I'm driving different tracks in Ableton using skeleton tracking to activate different virtual hotspot areas in a camera scene.

Facial Character Rigging

Here we rigged a 3D model up in maya to mimic my facial gestures. A talking hamburger, why not? The model was built by Mike Wharton at Superfad.

Face Swapping

Using openCV in Unity we can access the user's webcam, render it to a texture, and from that texture recognize facial features, extrapolating the relevant points in each detected face in order to swap one face with another.