Bilinear Interpolation is too slow on Python

[Suyash458]

I'm trying to correct the images from the controller using the code given here. The code runs fast on C++ but on Python it goes down to around 2 FPS. Is there any way to optimize this ?

C++ Code
Python Code

[Joe_Ward]

That's about the same as you would get using the warp() function, which isn't too surprising.

I've gotten images corrected using the GPU with a reasonable framerate, but haven't read back the textures, so I'm not sure if you might lose the performance gains there.

Another angle would be to find a Python library that can do the interpolation, possibly: SciPy interp2D or OpenCV remap. I haven't tried this, though. Note that the distortion map rarely changes, so if it takes a bit of time to setup the interpolator, you don't have to do it for every image.

[Suyash458]

Thanks!
I'll try using the interpolation libraries you've mentioned. If that's not of much help I guess pyOpenGL is the only way.

[Joe_Ward]

OpenCV's remap() function does it pretty fast. The image in the following example is upside down, but I'll leave fixing that as an exercise for the reader:

from __future__ import division
import os, platform, sys, threading
#Set this to the location of your Leap SDK
LEAP_SDK_PATH = os.path.abspath('/Users/jward/Desktop/LeapSDK')
# Get the correct Leap library directory for platform/bit-depth
current_platform = platform.system()
if current_platform == "Windows":
    arch_dir = 'lib/x64' if sys.maxsize > 2**32 else 'lib/x86'
elif current_platform == "Darwin":
    arch_dir = 'lib'
elif current_platform == "Linux":
    arch_dir = 'lib/x64' if sys.maxsize > 2**32 else 'lib/x86'
# Add that library to module search path
sys.path.insert(0, os.path.abspath(os.path.join(LEAP_SDK_PATH, arch_dir)))
import cv2, Leap, math
import ctypes
import numpy as np
destinationWidth, destinationHeight = 640, 240
destination = np.empty((destinationWidth, destinationHeight), dtype = np.ubyte)
def initDistortionMap(image):
    distortion_length = image.distortion_width * image.distortion_height
    xmap = np.zeros(distortion_length/2, dtype=np.float32)
    ymap = np.zeros(distortion_length/2, dtype=np.float32)
    for i in range(0, distortion_length, 2):
        xmap[distortion_length/2 - i/2 - 1] = image.distortion[i] * image.width
        ymap[distortion_length/2 - i/2 - 1] = image.distortion[i + 1] * image.height
    xmap = np.reshape(xmap, (image.distortion_height, image.distortion_width/2))
    ymap = np.reshape(ymap, (image.distortion_height, image.distortion_width/2))
    #resize the distortion map to equal desired destination image size
    expanded_xmap = cv2.resize(xmap, (image.width, image.height), 0, 0, cv2.INTER_LINEAR)
    expanded_ymap = cv2.resize(ymap, (image.width, image.height), 0, 0, cv2.INTER_LINEAR)
    return expanded_xmap, expanded_ymap
def interpolate(image, xmap, ymap):
    #wrap image data in numpy array
    i_address = int(image.data_pointer)
    ctype_array_def = ctypes.c_ubyte * image.height * image.width
    # as ctypes array
    as_ctype_array = ctype_array_def.from_address(i_address)
    # as numpy array
    as_numpy_array = np.ctypeslib.as_array(as_ctype_array)
    img = np.reshape(as_numpy_array, (image.height, image.width))
    #remap image to destination
    destination = cv2.remap(img, xmap, ymap, interpolation = cv2.INTER_LINEAR)
    #to resize output:
    #destination = cv2.resize(destination, (destinationWidth, destinationHeight), 0, 0, cv2.INTER_LINEAR)
    return destination
def process(controller):
    mapInitialized = False
    while(True):
        frame = controller.frame()
        image = frame.images[0]
        if image.is_valid:
            if not mapInitialized:
                left_x_map, left_y_map = initDistortionMap(frame.images[0])
                right_x_map, right_y_map = initDistortionMap(frame.images[1])
                mapInitialized = True
            undistorted_left = interpolate(image, left_x_map, left_y_map)
            undistorted_right = interpolate(image, right_x_map, right_y_map)
            cv2.imshow('Left Camera', undistorted_left)
            cv2.imshow('Right Camera', undistorted_right)
            if cv2.waitKey(1) & 0xFF == ord('q'):
                break
def main():
    controller = Leap.Controller()
    controller.set_policy_flags(Leap.Controller.POLICY_IMAGES)
    try:
        process(controller)
    except KeyboardInterrupt:
        sys.exit(0)
if __name__ == '__main__':
    main()

(edited to flip calibration map and remove erroneous image resize).

[Suyash458]

Thanks a lot! That was really helpful.
I'm using numpy.flipud to fix the image. Is there a better way to do it ?

[Suyash458]

Changing the indices in the for loop works as well.

def initDistortionMap(image):
    distortion_length = image.distortion_width * image.distortion_height
    xmap = np.zeros(distortion_length/2, dtype=np.float32)
    ymap = np.zeros(distortion_length/2, dtype=np.float32)
    for i in range(0,distortion_length,2):
        xmap[distortion_length/2 - i/2 - 1] = image.distortion[i] * destinationWidth
        ymap[distortion_length/2 - i/2 - 1] = image.distortion[i + 1] * destinationHeight
    xmap = np.reshape(xmap, (image.distortion_height, image.distortion_width/2))
    ymap = np.reshape(ymap, (image.distortion_height, image.distortion_width/2))
    #resize the distortion map to equal desired destination image size
    expanded_xmap = cv2.resize(xmap, (destinationWidth, destinationHeight), 0, 0, cv2.INTER_LINEAR)
    expanded_ymap = cv2.resize(ymap, (destinationWidth, destinationHeight), 0, 0, cv2.INTER_LINEAR)
    return expanded_xmap, expanded_ymap

[Joe_Ward]

I noticed that when using cv2.remap(), the calibration maps have to be the same size as the source image -- not the destination image. If needed, you could still resize destination after the remap operation. I edited the code above to make this clear.