Frames

LeapJS works by connecting to a websocket server running locally through the Leap Service. As often as every 10ms, the service sends a frame over the websocket and to the open page. This frame is a blob of JSON data including the positions of hands and fingers.

The browser only repaints the page every 16ms (60Hz) at best, and so these deviceFrames are not used directly. Instead, a LeapJS controller generates animation frames based off of the browser's "animation loop". (Using Node? Read up on the Javascript Frame Detection Loop)

The easiest way to access frame data is with the Leap Loop.

<div id=output></div>
<script>
  var output = document.getElementById('output');
  Leap.loop(function (frame) {
    output.innerHTML = 'Frame: ' + frame.id;
  });
</script>

ProTip™ — Although we're big fans of console.log here at Leap, it's actually not recomended while printing out frame data, as that can flood your console pretty quickly. For relaxing times, try instead setting the contents of an element on the page. You don't event need to quote your attributes.

Frame Objects

Every frame contains a lot of stuff, but the basic structure of what's needed most of the time is pretty simple. Here's the essence of a frame:

// Don't copy-paste this, it's not actually code.
LeapFrame = {
  hands: [
    {
      palmPosition: [x,y,z]
      palmNormal: [x,y,z]
      direction: [x,y,z]
      roll()
      pitch()
      yaw()
      pinchStrength: 0-1
      grabStrength:  0-1
      fingers: [
        {
          tipPosition: [x,y,z]
          direction: [x,y,z]
          bones: [
            {
              prevJoint: [x,y,z]
              nextJoint: [x,y,z]
              length: mm
              width:  mm
              center()
              matrix()
            }
          ]
        }
      ]
    }
  ]
}

Fingers and bones can be accessed through their names - thumb through pinky, and carpal, proximal, medial, and distal.

There actually a lot more that is available, such as length, speed, timeVisible, rotationMatrix, and so on. Check out the docs on Hand and Pointable for the details.

Here's an image that will be useful later. Maybe you should even save it to disk now.

Example

Enough talk! Let's write an app that moves cat based on how you move your hand. We want it to move left/right and up/down on the screen as your hand moves above the Leap, and to tilt when your hand tilts.

Here's how we'll break it down:

• An HTML image tag for the cat
• CSS absolute positioning to place the cat on the screen
• CSS Transforms to rotate the cat
• Leap screenPosition plugin to know where it should be placed
• Leap hand.roll() to tilt the cat.

Architecture

<script src="//js.leapmotion.com/leap-0.6.4.js"></script>
<script src="//js.leapmotion.com/leap-plugins-0.1.11.js"></script>

This lays out the structure of our application. We start with the `cats` data-store, a humble javascript object.

var cats = {};

Next we set up the loop. On every frame, look at every hand using the native javascript forEach method, and position the cat accordingly. setTransform() is stubbed out with no arguments, until we know what we need.

Leap.loop(function(frame) {
  frame.hands.forEach(function(hand, index) {
    var cat = ( cats[index] || (cats[index] = new Cat()) );
    cat.setTransform(...);
  });
})

We set up our Cat class, which will handle the DOM elements & interactions.

var Cat = function() {
  var cat = this;
  var img = document.createElement('img');
  img.src = 'https://s3-us-west-2.amazonaws.com/s.cdpn.io/109794/cat_2.png';
  img.style.position = 'absolute';
  img.onload = function () {
    document.body.appendChild(img);
  }
  cat.setTransform = function() {
    // todo
  };
};

Finally, we'll be using a LeapJS plugin to handle the conversion from coordinates in the real world to a position on the screen. Plugins run one after another before the frame is given to your application, allowing them to augment the frame with additional data before it gets used.

Manipulation

ProTip™ — Manipulate, noun: "to move or control something with your hands".

The screenPosition() plugin returns an array of numbers [x,y,z] of where the hand is on-screen in pixels. It does this by taking hands `palmPosition`, given in mm relative to the Leap Device, multiplying by a constant `scale` factor, and adding a constant `verticalOffset`.

In order for this to be available on the hand object, we have to tell the loop to use plugin. We do that by calling use, and passing in a custom scale factor.

roll() describes the motion your hand makes while turning a doorknob. It returns a number in radians from your palm facing downwards. Following the right-hand-rule with the z-axis coming towards us, we know that it is positive in the counter-clockwise direction.

Leap.loop(function(frame) {
  frame.hands.forEach(function(hand, index) {
    var cat = ( cats[index] || (cats[index] = new Cat()) );
    cat.setTransform(hand.screenPosition(), hand.roll());
  });
}).use('screenPosition', {scale: 0.25});

On the CSS side of things. We look at the first two values of the position array, x and y; and use them to position the image. We're careful to subtract the image dimensions, so that it stays on-center.

CSS transforms make quick work of the rotation value; and we take care to keep things cross-browser compatible. (css3please.com can help).

cat.setTransform = function(position, rotation) {
  img.style.left = position[0] - img.width  / 2 + 'px';
  img.style.top  = position[1] - img.height / 2 + 'px';
  img.style.transform = 'rotate(' + -rotation + 'rad)';
  img.style.webkitTransform = img.style.MozTransform = img.style.msTransform =
  img.style.OTransform = img.style.transform;
};

That's almost it! Let's add a little shim so that the cat shows up even without the Leap.

// inside the Cat definition:
  img.onload = function () {
    cat.setTransform(
      [
        window.innerWidth/2,
        window.innerHeight/2
      ],
      0 );
    document.body.appendChild(img);
  }
// At the tail-end of the script
cats[0] = new Cat();

Putting it all together

var cats = {};
Leap.loop(function(frame) {
  frame.hands.forEach(function(hand, index) {
    var cat = ( cats[index] || (cats[index] = new Cat()) );
    cat.setTransform(hand.screenPosition(), hand.roll());
  });
}).use('screenPosition', {scale: 0.25});
var Cat = function() {
  var cat = this;
  var img = document.createElement('img');
  img.src = 'https://s3-us-west-2.amazonaws.com/s.cdpn.io/109794/cat_2.png';
  img.style.position = 'absolute';
  img.onload = function () {
    cat.setTransform([window.innerWidth/2,window.innerHeight/2], 0);
    document.body.appendChild(img);
  }
  cat.setTransform = function(position, rotation) {
    img.style.left = position[0] - img.width  / 2 + 'px';
    img.style.top  = position[1] - img.height / 2 + 'px';
    img.style.transform = 'rotate(' + -rotation + 'rad)';
    img.style.webkitTransform = img.style.MozTransform = img.style.msTransform =
    img.style.OTransform = img.style.transform;
  };
};
cats[0] = new Cat();

Next Steps

At this point, you know how to make fully-qualified a Leap app. But there are plenty of bells and whistles to keep you going.

Controllers

You should know that under the hood, Leap.loop is running a controller. Any time you need to, a controller can be set up and used without the loop. Remember to call connect().

var controller = new Leap.Controller({enableGestures: true})
  .use('screenPosition')
  .connect()
  .on('frame', function(frame){
    // Try making some circles
  })

Backgrounding

You may notice that at the bottom of the codepen example itself, there is an extra line. By default, LeapJS stops animating when a windows is out of focus, in order to save CPU usage. This functionality can be easily disabled:

controller.setBackground(true);

Further Resources

We love to hear from you! Visit the community forums for help or to share your work, or visit the issue tracker on GitHub if you've got feedback on the library itself.