The Leap Motion controller uses infrared stereo cameras as tracking sensors. The LeapC API provides an image buffer containing the sensor brightness values and a distortion buffer containing the camera calibration map. The distortion map can be used to correct lens distortion in the image data. The function LeapRectilinearToPixel() can be used to undistort specific pixels in an image and the function LeapPixelToRectilinear() can be used to find the 3D location corresponding to a matched pair of stereo pixels.

Requesting Images

Images are not sent automatically. For each desired stereo image pair, you must call LeapRequestImage(), identifying the associated tracking frame and supplying a buffer large enough to hold both stereo images. You must request a set of images within a few frames of the corresponding tracking frame, otherwise the image data will have already been discarded.

When the image is available, it is written to the supplied buffer. Once the buffer is completely written, a LEAP_IMAGE_COMPLETE_EVENT is added to the event queue and can be accessed via LeapPollConnection(). The image complete event contains image properties, distortion correction data, and a pointer to the buffer.

Both images from the stereo cameras are written to the same buffer; left image first and then the right image.

To get an image:

Open the connection.
Poll for tracking frames.
Allocate a buffer large enough to hold both pairs of the stereo image.
Create a LEAP_IMAGE_FRAME_DESCRIPTION struct.
Set the struct fields to identify the desired image frame and type and specify your image buffer.
Create a LEAP_IMAGE_FRAME_REQUEST_TOKEN struct.
Call LeapRequestImages(), passing in the connection handle, the description struct and the token struct.
Save the token struct. If you need to deallocate your image buffer before the request is complete, you must call LeapCancelImageFrameRequest() with the token first. Otherwise, LeapC could write to the deallocated memory area. The token also correlates the image complete message with the original request.
Continue polling for tracking frames and status messages. When the image request is complete and the buffer has been written, a LEAP_IMAGE_COMPLETE_EVENT message is provided by LeapPollConnection(). The LEAP_IMAGE_COMPLETE_EVENT struct contains the image token, a description of the image, and the distortion map.
After the image is complete, you can release or reuse the image buffer when desired.

The following example illustrates how to call the LeapRequestImages() function:

 LEAP_IMAGE_FRAME_DESCRIPTION frameDescription;
 frameDescription.type = eLeapImageType_Default;
 frameDescription.frame_id = frame->info.frame_id; //frame is LEAP_TRACKING_EVENT struct
 frameDescription.buffer_len = image_size; //the size of image_buffer
 frameDescription.pBuffer = image_buffer; //typically allocated in advance based on known image size
 
 image_token = malloc(sizeof(LEAP_IMAGE_FRAME_REQUEST_TOKEN));
 eLeapRS result = LeapRequestImages(*connection, &frameDescription, image_token);
 if(result != eLeapRS_Success)
   printf("LeapRequestImages call was %s.\n", ResultString(result));

If the image buffer is too small, the image request will fail. Instead of an image complete event, you will get an image request error event. This error event will contain the required buffer size, which you can use to re-allocate your buffer for subsequent image requests.

Determining the Image Buffer Size

The required image buffer size depends on the device type as well as the image type and format. The buffer size depends on the width, height, and pixel format (bytes per pixel) of the image. A single buffer must hold stereo images. For example, if the current images produced by a Leap Motion device is 640x240, 1-byte pixels, then the buffer size must be: 640 x 240 x 1 x 2 = 307,200 bytes. However, since the resolution can change dynamically, your code should be capable of resizing the image buffer as needed.

If your image request supplies a buffer that is too small for the current image, LeapC provides an error message via LeapPollConnection(). The LEAP_IMAGE_FRAME_REQUEST_ERROR_EVENT struct contains the required buffer size for the requested image type and format. You can use the required_buffer_len field to reallocate the buffer for future image requests. (The image request that contained the insufficient buffer is not fulfilled – you would need to re-issue the request with an updated buffer to get that particular image. For most purposes, though, it is acceptable to skip a failed request and just continue on with the next frame's image.)

The following example code resizes the byte array used to hold the image data based on the required_buffer_len field of an LEAP_IMAGE_FRAME_REQUEST_ERROR_EVENT struct:

 //imageErrorEvent is a LEAP_IMAGE_FRAME_REQUEST_ERROR_EVENT struct
 //Resize image buffer if too small
 if(image_size < imageErrorEvent->required_buffer_len){
   image_size = imageErrorEvent->required_buffer_len;
   if(image_buffer) free(image_buffer); //image_buffer is an array of byte
   image_buffer = malloc((size_t)image_size);
   printf("Resized image buffer to %lli.\n", (long long int)image_size);
 }
 
 if(imageErrorEvent->error == eLeapImageRequestError_ImagesDisabled)
   printf("Warning: Images disabled. Check your control panel settings.");

Image Distortion

When a ray of light enters one of the Leap Motion cameras, the lens bends the ray so that it hits the sensor, which records it as a greyscale brightness value at a specific pixel location. Of course, no lens is perfect, so a ray of light does not land on the sensor in the optically perfect spot. The distortion map provides data to correct this imperfection, allowing you to calculate the true angle of the original ray of light. You can use the corrected angle to generate a rectified image, and, using the angles from both images in the stereo pair, you can triangulate the 3D location of a feature identified in both images.

For image rectification, the distortion map can be fed to a shader program that can efficiently interpolate the correction applied to rays of light to produce a texture containing the rectified image. For getting the true angle for a small set of points, you can use the LeapPixelToRectilinear() function (but this is not typically efficient enough to transform a full bitmap at a high frame rate).

Rectifying Image Points

To rectify individual points in an image, you can call the LeapRectilinearToPixel() function. Given a ray from the camera, this function returns the pixel coordinates in the image buffer that contain the light recorded from that direction, corrected for lens distortion.

The following code takes a target texture size and determines the correct pixel brightness value. For each pixel, the code computes the ray direction to the point in the scene that would illuminate the target pixel given an ideal optical system. The code then corrects for optical distortion by calling LeapRectilinearToPixel(), which provides the pixel coordinates in the image buffer that actually contains the brightness value for the target pixel. The code copies the brightness value to the target texture buffer. The result is a rectified image.

 //The pixel size of the textures we write the undistorted images to
 #define TEX_WIDTH 400
 #define TEX_HEIGHT 400
 //Max field of view varies by device, use 8 for the peripheral, 22 for Rigel
 #define MAX_FOV 22
 
 //image_buffer contains the image data
 for( float row = 0; row < TEX_HEIGHT; row++ ) {
   for( float col = 0; col < TEX_WIDTH; col++ ) {
     //Normalize from pixel xy to range [0..1]
     LEAP_VECTOR input;
     input.x = col/TEX_WIDTH;
     input.y = row/TEX_HEIGHT;
 
     //Convert from normalized [0..1] to ray slopes
     input.x = (input.x - .5) * MAX_FOV;
     input.y = (input.y - .5) * MAX_FOV;
 
     LEAP_VECTOR pixel = LeapRectilinearToPixel(*connection, eLeapPerspectiveType_stereo_left, input);
     int dindex = (int)floor(row * TEX_WIDTH + col);
     int pindex = (int)roundf(pixel.y) * image_width + (int)roundf(pixel.x);
     if(pixel.x >= 0 && pixel.x < image_width && pixel.y >=0 && pixel.y < image_height){
       undistorted_image_left[dindex] = ((char*)image_buffer)[pindex];
     } else {
       undistorted_image_left[dindex] = 128;
     }
   }
 }

For the right image, remember to offset the index into the image_buffer array since the right image follows the left.

You can get the actual device field of view from the LEAP_DEVICE_INFO struct. Use tan(fov/2) in the calculation above instead of MAX_FOV.

Note that LeapRectilinearToPixel() may not be fast enough to rectify the image to a high resolution texture in real-time. Generally you should limit its use to individual points, image patches, or low-resolution textures. For better performance for full images, you can use a shader to rectify the image.

See Rectifying Points Example

Rectifying with a Shader

To rectify an entire image with a shader, create a texture containing the data in the distortion buffer. The distortion buffer is contained in the distortion_matrix field of the LEAP_IMAGE_COMPLETE_EVENT struct. This data is set up to use a shader's normal uv interpolation mechanism. You can use the interpolated uv coordinates to look up the corrected brightness in the image texture, as in the following fragment shader program:

 uniform sampler2D rawTexture;
 uniform sampler2D distortionTexture;
 
 void main()
 {
   vec2 distortionIndex = texture2D(distortionTexture, gl_TexCoord[0].st).xy;
   float hIndex = distortionIndex.r;
   float vIndex = distortionIndex.g;
 
   if(vIndex > 0.0 && vIndex < 1.0 && hIndex > 0.0 && hIndex < 1.0)
   {
     gl_FragColor = vec4(texture2D(rawTexture, distortionIndex).rrr, 1.0);
   }
   else
   {
     gl_FragColor = vec4(0.2, 0.0, 0.0, 1.0);
   }
 }

To interpolate correctly, the distortion texture must use the following filter and wrap parameters:

 glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
 glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
 glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
 glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

The distortion texture itself must use two floating point values per texel and contain 64x64 texels:

1 glTexImage2D(GL_TEXTURE_2D, 0, GL_RG, 64, 64, 0, GL_RG, GL_FLOAT, distortion_buffer_left);

The distortion map itself rarely changes – the only time it changes is if the Leap Motion hardware is swapped with a different device or if the device is rotated so that the user's hands enter from the opposite side of the vertical field of view. In this case the Leap Motion software inverts the image, and the distortion map, to automatically maintain the proper orientation. (Auto-orientation can be turned off in the Leap Motion service configuration.)

See Rectifying with a Shader Example

Detecting when the Distortion Map Changes

The distortion map contains a grid of values that can be used to rectify the images. Rectification should be performed if you are aligning 3D objects with pixels in the images or if you are performing stereo triangulation on the image pair.

Each image complete event contains the distortion map for that image, however, since translating the map to an interpolation array or shader texture can itself be an expensive task, you typically only want to do this when the distortion map actually changes – not once for every image request. The distortion map can only change when the images are flipped due to automatic or manual re-orientation or if the device itself is changed. (For internal reasons) there is no way to tell that a reorientation has occured. However, you can use the LEAP_IMAGE_COMPLETE_EVENT::matrix_version field to detect when the distortion map changes. When the matrix_version changes between images, you know that the distortion map has changed. The matrix version number ALWAYS increases when the distortion map changes and never returns to a prior value.