What is a light probe?

In order to successfully composite CG elements into live-action scenes it is important that the lighting of the object match the lighting of the scene. One technique for reproducing the incident light in a live-action scene is to create a high dynamic range photograph of a mirrored ball placed in the scene (called a “light probe”) and then use that light probe as a source for image-based lighting.

Currently, in order to create a high dynamic range image of a mirrored ball, one must take a series of photographs with the exposure value of each image being stopped down by a given increment from the exposure value of the one before. Later, each of the images are assembled into a single high dynamic range image using a program such as MakeHDR.

Why a real-time light probe?

If an artist wished to accurately illuminate a CG object traveling through a complex lighting environment, it would be necessary to capture light probes at numerous locations (ideally at every frame) along the object’s path. Clearly, this would be an ambitious task.

How did we solve this?

One solution for creating a real-time high dynamic range light probe is to develop a system in which multiple exposures of the same image can be captured within a single video frame. We did this by modifying a five-point Multi-Image Filter (i.e., a faceted lens that is commonly used to create photographic kaleidoscope effects), and applying successively increasing values of neutral density gel to four of the five facets of the filter (3⅓, 6⅔, 10 and 13⅓ stops). This modified filter effectively produces a single image that is divided into five identical regions, with the center region capturing a “direct” view and the four outer regions stopped down to their respective exposure values. This modified filter is placed on a video camera that is mounted along with a mirrored ball on a span of angle iron (see Figure 1)

Assuming the relation between the camera and the ball never changes, the light probe only needs to be calibrated once. To compensate for the angle shift introduced by parallax effects from the facets of the multi-image filter, one can compute the arctangent of the distance between facets divided by the distance between the lens and the silver ball. By determining the number of degrees each facet is offset from the center, we are able to warp each region of the filter according to the direction space of its view of the ball. In our case, each facet’s view of the ball was computed to be 2.7 degrees off from center.

More accurate calibration can be done with the help of a light stage, which provides a “master key” for factoring out lens distortion and imperfections in the mirrored ball. However, we found that simply computing the pixel shift and then overlapping each region of the filter was sufficient for assembling a usable image.

In order to capture high dynamic range light probe data at every frame along a path, one presses ‘record’ on the video camera and carries the light probe along the desired path. A computer program then imports each recorded frame, isolates the five distinct images in the frame, aligns them according to predetermined calibration data, and then assembles the aligned images into a high dynamic range omnidirectional measurement of incident illumination.

This new technique will permit artists to composite CG objects into dynamic complex lighting environments, accurately reproducing high dynamic range lighting parameters for each frame. In the future, this technique would benefit from greater precision in applying the neutral density gels to the multi-image filter, a smaller camera rig, and higher resolution video cameras.

I presented this project at SIGGRAPH 2001 as a technical sketch. Read it here.

The University of Southern California applied for a patent for this system. Read the patent application here.