Mike Heavers

I'm fascinated by how a machine sees the world. It's capability for doing things like perceiving detail and detecting patterns far surpasses ours, but its ability to infer context and meaning is only as good as the education we give it. As a result, what it dreams up runs the gamut from bungled to brilliant, but unquestionably offers enormous potential to aid in our own ability to interpret and more deeply explore what's around us.

ROLE: computer vision, machine learning, 3D rigging, sound design, protocols

TECH: openCV, Unity, Kinect, C#, C++, React Native, Pix2Pix, Tensorflow, Ableton Live, openFrameworks

Face Detecting Selfiecam

Senses phone stabilization and detects faces to automatically snap photos so you don't have struggle with that arm's length button press stuff.

Landmarker

A P5.js tool to generate facial landmarks for use in Computer Vision experiments where facial landmarks cannot be detected for a given image. Particularly useful for non-human faces in facial tracking and facial transformation applications.

Machine Learning Driven Photo Background Replacement

Using semantic image segmentation in Tensorflow to automatically separate people from their backgrounds in photos, as a prototype alternative to green-screening / chroma-keying.

Head Tracking as a Controller

We rely so heavily on our hands for controlling games, I think we often forget there are often more intuitive interfaces at our disposal. Here I'm using the Kinect to control a character in Unity. His head movements mimick mine, and when I squint, the camera zooms in.

Controlling Audio with Facial Tracking

Another intuitive use of the facial tracking I can imagine would be to drive audio. Our mouths take very specific shapes to produce specific sounds and accents. I can imagine an application where facial tracking is used to give voice to animated characters, or as I'm experimenting with this video, mapping different features to different audio characteristics, such as pitch / frequency.

Controlling Music with Skeleton Tracking

Music is so tied to body movement, why not let your movements control the music? Imagine a DJ driving sound not through a giant MIDI controller, but through the movements of their body. While my dance moves don't exactly measure up, here I'm driving different tracks in Ableton using skeleton tracking to activate different virtual hotspot areas in a camera scene.

Facial Character Rigging

Here we rigged a 3D model up in maya to mimic my facial gestures. A talking hamburger, why not? The model was built by Mike Wharton at Superfad.

Face Swapping

Using openCV in Unity we can access the user's webcam, render it to a texture, and from that texture recognize facial features, extrapolating the relevant points in each detected face in order to swap one face with another.

Face Substitution

Mapping significant facial features between the camera feed and features detected in the photograph.

Face Smashing

Distorting the parameters of the feature mappings