Developer Labs

Touchless rocket design, robotic controls, massive art exhibits, live musical performances – it’s been an amazing year at Leap Motion. When we launched on July 22, we could scarcely imagine how our developer community would be able to use our SDK to build the next generation of 3D apps.

Now, as 2013 draws to a close, we’re taking a look back at some of the coolest videos and most disruptive projects from the year. While many of our community’s greatest achievements are fully realized apps in the Airspace Store, here are the outstanding highlights and hidden gems of 2013.

From the earliest theremin experiments to the latest music apps in the Airspace Store, it’s been a banner year for live performances with the Leap Motion Controller. Musicians have flocked to apps like GecoMIDI to interact directly with music streams and create whole new sounds.

Bringing touchless interaction to live performance. Musicians around the globe have used the Leap Motion Controller to tweak and transform music as it’s being performed – including a classical piano, invisible drums, beatboxing, dubstep, acoustic guitar, and a combined theremin and effects machine. Tokyo-based industrial artist Aliceffekt launched the first known album created with the Leap Motion Controller.

Can one person play an entire live orchestra with only two hands? Hagai Davidoff was able to dynamically control the flow of a full classical ensemble through natural hand and finger movements.

Explore a universe of music. Leap Motion engineer Isaac Cohen’s Universe of Sound is a 3D space-exploration music blog that links sound and visuals for a transcendent journey through the stars.

Changing the face of game design. From ergonomics and the keyboard, to visual feedback and smart control design, you can uncover some key insights within four Airspace games and M1 Interactive’s post on moving beyond traditional controls.

Art follows tech – prototyping with Leap Motion. On the popular game industry blog Gamasutra, Double Fine developers Patrick Hackett and Drew Skillman presented a case study about their game Dropchord, showing how experimenting with a new technology can lead to fundamentally new game concepts.

Encountering new technology is like a clash of cultures. Appearing at last month’s App Developers Conference, Funktronic Labs founder Eddie Lee talked about the future of developing apps in the third dimension, and how developers can design intuitive interactions with user uncertainty in mind.

Elon Musk’s rocket part prototyping. In August, we saw how SpaceX engineers are integrating the Leap Motion Controller with their CAD platform to manipulate virtual rocket parts – diving into virtual models and pulling them apart to better visualize how they work.

Pushing the boundaries of interaction design. Overturning old design philosophies to explore new concepts – it’s what we love to do. Our recent features on menu design prototyping and touchless recipe browsing are just the latest stages in our ongoing exploration of interaction concepts. Some of the most exciting projects in 2013 included M1 Interactive’s Unity starter kit, Rob O’Leary’s web-based motion interface, and Theo Armour’s work with LeapJS.

Reaching into 3D design with Autodesk. We live in a designed world – where form and function are woven together into our everyday lives. With Autodesk Maya and the Leap Motion Controller, industry professionals are now pushing past the flat interface to directly design and manipulate models. 2013 was just the beginning as developers continue to explore the best ways to reach into virtual spaces.

Technology in the operating room can be a double-edged sword. It lets doctors and nurses quickly access medical images and patient data, but maintaining sterile gloves and surfaces to avoid spreading infections can be time-consuming.

This year, doctors and surgeons around the world experimented with touchless interaction to avoid touching technology under sterile conditions. Trials have already begun with TedCas’ medical imaging software in 6 hospitals and 2 medical research centers in Ireland, Spain, Argentina, Chile, Canada, and the United States.

From holograms and illusions to unusual projections, developers have been using the Leap Motion Controller to bring a little magic and mystery to 3D displays. M1 Interactive created a transparent LCD display that lets you reach out and rotate a virtual tarbosaurus skull. You can dismantle and reassemble a Pepper’s ghost car with Integrity’s interactive hologram, or cast your vote for your favorite book by pointing with DOKLAB’s voting station.

We live in an ocean of data that is mostly only accessible through flat interfaces. That time is coming to an end, and Leap Motion developers are at the forefront – exploring new ways to reach into data and control it.

Rob O’Leary created a 3D graphing projection that combines Leap Motion interaction, voice control, and semantic search. Leap Motion experience engineer Isaac Cohen built a data pool that reveals just four of the Leap Motion Controller’s parameters. Plus, Mike Brondbjerg’s air data visualization project lets you navigate through the 3D mesh of London’s air pollution.

This month’s RobotsConf sparked a variety of scripts and hacks (Sphero, anyone?), but they’re just the latest in a long line of Leap Motion-driven robotic projects – covering robots of all shapes, sizes, and abilities.

Bridging the gap between biological and robotic hands has inspired developers to create a moveable arm/crane, a 3D-printed hand, and an arm with 5 axes of movement. One of the most intriguing videos was Mutasim Ali’s time-lapse construction video – taking us from tinkering to testing a robotic arm built on a remote-control car.

NASA engineers at this year’s Game Developers Conference demonstrated how the Leap Motion Controller can be used to remotely control a one-ton, six-legged space rover. The ATHLETE’s baby brother, a hexapod, was also integrated with the Leap Motion Controller, thanks to Queron Williams and Processing.

Finally, Niels Jacot’s interface for Nao – a two-foot-tall humanoid robot with articulated arms, legs, and head – let him control different parts of the robot’s body, and even let him walk.

What happens when you bring a Leap Motion Controller into a classroom? Special needs teacher Mathieu Marunczyn was one of the first to explore how Leap Motion interaction can help children with autism connect with apps – and each other.

Foreign language immersion. Step into a special room at Ohio University’s Athens campus to travel virtually anywhere on Earth. Besides exploring cities with Google Street View, this experimental “Holodeck” has been used for everything from an IMAX style chalkboard to Chinese calligraphy practice and Minecraft.

2013 was the year of the disappearing interface. Leap Motion interaction spread from the world’s first computer with embedded Leap Motion technology to 12 new platforms and a 3D interactive keyboard, while developers have been working to bridge the gaps between hardware, software, and human beings in new ways.

A new perspective with Oculus Rift. Gerald Terveen’s Oculus integration, John Selstad’s experiment videos, and our immersive Google Earth experience at May’s Google I/O conference are just the beginning, as Leap Motion developers continue to experiment with this budding new technology.

Smart homes and the Internet of Things. Want to control colors with a wave of your hand or control your whole house? Watch in 2014 for a variety of touchless apps that let you reach out and take control of your environment.

Create and print 3D designs. Being able to create something in thin air and see it appear before your eyes used to be the stuff of science fiction. Hot Pop Factory recently brought this experience to visitors at Toronto’s Mini Maker Faire, who created 3D drawings that came to life as plastic sculptures. Now anyone can access this ability with the new clay sculpting app Freeform, which lets you save your designs in 3D-printer-friendly formats.

Touchless interaction adds a new dimension to art installations – both literally and figuratively. The hidden quality of interacting with thin air draws attention away from the interface, letting viewers feel immersed in the art itself. Here are four stunning installations that made waves this year:

Reach into a surreal world of alien shapes. Untitled° by NATURE graphique allowed viewers to immerse themselves within a monochromatic 3D scene, using the Leap Motion Controller to explore the landscape with simple hand movements.

Human actions, natural consequences. Growth is a 3D-projection mapping experience that lets you manipulate digital vines, branches, and beams of light cropping up against a stark blue sky. Soothing movements create beautiful scenes, while aggressive swipes summon darkness – telling the story of humanity’s relationship with nature.

What if social media was art? Created by Forge Collective, Connexion Point allowed complete strangers to make connections and create a dynamic digital community – exploring stories through Leap Motion interaction.

Welcome to our universe; have a seat. Created in WebGL at B-Reel’s London office, Star Canvas is an interactive guestbook in the form of a giant projected star chart.

Wherever our platform evolves in the future, the creativity and inventiveness of our developer community has played a major role in shaping how we think about 3D interaction. Everyone who has built an app, experimented with our API, or answered a question on the forums has helped to push the Leap Motion Controller to new heights.

What do you think of this year’s top projects? Did we leave anything out? What would you like to see in 2014? Let us know in our 2013 retrospectacular forum thread.

Isaac Cohen is an experience engineer with Leap Motion.

The process of instilling wonder has always fascinated me. It’s such an indescribable emotion, but so fervent and real. Attempting to make a person feel wonder is a marvelous quest.

In this post, I’d like to examine a project I worked on, called The Universe of Sound. It’s something that I’ve been working on for quite some time, and although it probably isn’t as cool as I hope to make it sound, I am proud of it, mostly because of how much of my life it consumed.

In my experience, inspiration rarely comes when I call, and even more rarely takes form in logical, coherent thoughts. My mantra is always to create as much as possible and see what takes shape. Here’s the demented process that was the creation of the Universe of Sound.

Read More

Mischa Spiegelmock is a software engineer and architect with a variety of specializations, including security, web applications, and real-time media. One of the earliest members of our developer program, he’s published a book about Leap Motion development essentials. You can follow Mischa’s latest work on GitHub.

Like many people, as soon as I saw the first demonstration video of the Leap Motion Controller, I knew I had to play with one. I pre-ordered a developer kit, and finally got my hands on one in September 2012.

Read More

Hi, I’m Hagai Davidoff. I specialize in music for theater, film, media and MIDI mockups. Recently, I performed a full orchestra using just a keyboard, orchestral VSTIs, and GeoMIDI with the Leap Motion Controller. Check out the full video below, or skip to 6:00 to hear the music.

My main passions are orchestral music and music technology, and I love to create acoustic simulations. Over the course of my daily work, I always try to find the most natural way to input MIDI data into my digital audio workstation (DAW). That’s why my studio is filled with different kinds of MIDI gadgetry and controllers – pad controllers, MIDI keyboards, breath controllers, various iPhone apps, etc.

I’ve always felt a huge gap between hardware and software, the latter being far superior in terms of possibilities and details. My MIDI keyboard couldn’t reach all the velocity layers present in the sample libraries I use. I couldn’t naturally glide between different musical articulations intuitively (I hate keyswitches).

I could go on, but you get the idea. What good is a 20-velocity-layers sample library, if your keyboard always roughly transmits the same 10 velocity layers?

So I tried to re-organize my hardware life by buying an electric piano. These are usually much more responsive, yet costly. Since most digital pianos don’t come with knobs and faders, I’ve hacked my old Maudio Keystation Pro 88 into a MIDI control box and fused it with my breath controller.

Thus, I’ve built my perfect MIDI freak controlling extravaganza, which you can see in my full photoblog. Here’s my spaceship:

But there was still something missing. Tactile control has a price – resistance. Sometimes this resistance is a musical must, sometimes it just kills your natural (MIDI) curves.

When I first saw a video demo of the Leap Motion Controller, the obvious wow factor hit me. I immediately thought in musical terms and tried to figure out how I could implement Leap Motion into my workflow.

Then came GecoMIDI – the answer was there. The possibilities overwhelmed me, and I started experimenting. The result you see in the video at the top of this post is one of the first ideas I came up with, and already it feels very rewarding.

Suddenly, you can control XYZ with one open hand, and another set of XYZ with the same hand, closed. Now double that for two hands, and by the time I finish writing this, the tech boffins out there will probably make gestures of individual fingers a MIDI reality (go Geert go), taking things to a whole new level.

Hand gestures is an intuitive and much more natural way to create music. When I play my current setup, I feel like I’m dynamically conducting an orchestra. It helps me forget the techy stuff and get down to the musical Ideas and the joy of composing.

When you combine good hardware technology with excellent software (like GecoMIDI and the Spitfire Audio “Albion I” sample library, which I’ve used in the video), the results can be stunning.

There are still apps and features I would love to see from the developers out there in Leap Motion land:

1. A hand glide gesture that sends actual MIDI notes. This would enable me to create natural glisses for virtual harps (I know, I know, but AirHarp doesn’t send MIDI out).
2. An app that translates conducting into actual tempo tracking in my DAW (like a real conductor).
3. General better and more stable tracking.

I’ve never seen anyone use the Leap Motion Controller like this before, and I feel honored to be an early adopter of this creative technique.

Hagai Davidoff is a critically acclaimed composer, producer, and arranger specializing in orchestral and acoustical simulations. He also produced sample libraries for Sonokinetic, and is a Cubase, Production, and Virtual orchestration teacher at BPM College in Tel-Aviv, Israel. For more info and music, check out hagaid.com.