Dual Camera Infrared Chroma Key

Anyone who has shot with a green screen can tell you that chroma key is not a perfect system. You’re shooting a subject in front of a green wall and then telling a computer to discard any pixels that are greener than a certain amount.

Problems arise when green light bounces off the back wall and onto your subject – this is called “green spill”. Additional problems arise if your subject happens to be wearing green or has green eyes. Semi-translucent areas are trouble as well, such as frizzy hair, the thin parts of the ear, etc.

While working in Paul Debevec’s graphics lab at the University of Southern California’s Institute for Creative Technologies, it occurred to me that there ought to be a keying system that uses an invisible wavelength of light.

Step one was building a video camera that records red, green, blue, and IR light channels. I prototyped this by mounting two consumer-grade video cameras on a plank of wood, each positioned to look through an 80/20 beam splitter. One camera was set to capture a regular image of the subject, the other camera had a filter over the lens that blocked all visible light and only permitted IR light to pass through.

Step two was illuminating a reflective screen behind the subject with infrared light. Turns out you can buy IR light emitters from most security camera shops. They’re popular for lighting garages so security cameras can shoot in the dark.

Step three was shooting a subject in front of the screen with the two cameras.

Step four is where the magic happened. Load the two shots into After Effects, align them and set the IR channel as a traveling matte for the RGB. Add a background and voila. A perfect matte!

The infrared keying system is described in our SIGGRAPH 2002 Light Stage paper: https://dl.acm.org/citation.cfm?id=566614

Paul E. Debevec, Andreas Wenger, Chris Tchou, Andrew Gardner, Jamie Waese, Tim Hawkins (2002). A lighting reproduction approach to live-action compositing. Proceeding SIGGRAPH '02 Proceedings of the 29th annual conference on Computer graphics and interactive techniques. Pages 547-556