In many ways, waving your hands in space is a lot like moving a mouse around the screen – you’re limited to rather simple interactions until you can trigger a discrete event. In the case of the mouse, we can trigger events with clicks. But how can we trigger events with a touchless interface like the Leap Motion Controller?

In the video below, I outline and demonstrate several discrete gestures you can use to trigger events in your apps.

In this post, we’ll take a look at:

• position/location based triggers
• swiping/punching/poking gestures
• picking/plucking/finger-wiggling gestures
• flicking/pinching/grabbing gestures

Position and Location-Based Triggers

These involve placing or moving your hand into a specific location in 3D space. Users of the Kinect will be familiar with these methods for navigating menus. For instance, users can place a hand over an icon; after a second of holding it there, the icon is “clicked” or triggered.

Position-based triggering requires some external feedback, such as a visual overlay of the hands on the screen, and/or highlighting of the button. (Perhaps, with some user-specific calibration, internal body feedback would be enough; but that’s another discussion entirely).

Swiping Gestures

These include fast movements in one direction along any of the 3 axes. Physically, the swipe gestures make the most sense on the X and Y axes. Swiping gestures have been built into the SDK for a while now, and many of you have seen demonstrations of swiping gestures apps that use page turning.

Swiping gestures that move back and forth along one axis may work well for music applications, e.g. bowing or record scratching.

Punch, Poke, and Chop Gestures

These are much like swipe gestures, where there’s a change in one direction. Physically, punch gestures make sense on the Z axis, while chop gestures make sense on the X or Y axis.

Finger-Based Triggers

So far, the gestures I’ve discussed use palm position and delta data to trigger events, but fingers can also be used to trigger events. Computer and tablet users are already accustomed to triggering events via fingers when clicking a mouse, pressing keys, or poking a tablet.

Pick, pluck, or wiggle: This can be detected when the finger position changes suddenly while the palm data holds steady.

Grab or pinch gesture: We can understand the grab gesture as closing your hand by pressing your fingertips to your thumb. The inverse of pinching, what I would call “flicking,” is the sudden disconnect of your fingertips from your thumb.

Using Finger-Based Triggers in Music Apps

I have a special fondness for the pinch and flick gestures in triggering sound events (like MIDI noteOn and MIDI noteOff events). This is because they:

• are easy to track by measuring the distance between the fingers and the thumb, or by using the raw finger count on a given hand;
• have built-in tactile feedback – you can feel when your fingers have connected or disconnected with your thumb; and
• are especially satisfying when used in combination with palm position.

Other gestures such as swiping, chopping, punching, and poking can also be used in combination with other continuous position variables – such as the vertical position controlling pitch of the cello like instrument in the video above.

However, please keep in mind that there may be some two or three dimensional movement with a swipe gesture. A user could start in one position on the Y axis and be using the X axis to trigger, but the swipe may move at a 30 degree angle; thus, the Y position and the variable sent when triggered might be different than the intended starting point.

Another thing to keep in mind: take care when using the pinch gesture in combination with palm orientation (specifically roll), as the finger count data could be skewed when fingers mask each other from the Leap Motion Controller’s cameras.

Thoughts and Discussion

This collection of gestures is only a starting point, and there will be many more to consider as we continue to explore what’s possible with a gesture-based interface. But with each new trigger gesture, it’s important that we evaluate how that gesture (meant to trigger an event) will interact with any continuous palm position and orientation-based controls.

Let’s wrap this up with a discussion. What are your favorite trigger events? How have you used them in combination with continuous controls? How many combinations of gestures have you been able to have active at one time without cross-talk between them?