Writing your own applications
The hand-tracking server uses a simple network protocol to communicate with your applications over a socket. This means you can write your applications in any language you choose. We provide example libraries to communicate with the hand-tracking server in Java (API Docs), C++ (API Docs), and C# (API Docs), and describe how to use these libraries below. If your application is in a different language, you may need to implement our protocol in that language. (Don't worry, it's not too hard!)
Once connected to the server, the HandTrackingClient library starts its own thread, waiting for new messages (HandTrackingMessage). Received messages (e.g. Pressed, Dragged, Released, Moved, etc.) are passed to registered callbacks that implement the HandTrackingListener interface.
Each message contains the position and orientation of both hands. The coordinate space of the hand position is specified by the camera setup stage. For a standard camera setup, the x-axis points right; the y-axis points up; and the z-axis points away from the screen. Units are in millimeters and the origin is at the center of the checkerboard you used during calibration.
Getting started
The Java, C++, and C# examples are located in the NimbleSDK\SDK directory. Take a look at the README.txt file in the NimbleSDK\SDK directory for additional instructions on how to load the examples.
To run any of the example applications below, you'll need to have the hand-tracking server running. From the NimbleSDK directory, run:
nimble_server.bat
The simplest application (Java)
The "Echo" program (located in the NimbleSDK\JavaExamples\src directory) uses the Java HandTrackingClient library to communicate with the hand-tracking server. It registers a listener (HandTrackingListener) which is called-back with the messages received from the server.
package com.threegear.apps.demos;
import java.io.IOException;
import com.threegear.gloveless.network.*;
public class Echo {
public static void main(String[] args) throws IOException {
HandTrackingClient client = new HandTrackingClient();
client.addListener(new HandTrackingAdapter() {
@Override
public void handleEvent(HandTrackingMessage baseMessage) {
if (baseMessage instanceof PinchMessage) {
PinchMessage message = (PinchMessage) baseMessage;
System.out.printf("Received event %s for hand %s at position %s\n",
message.getType(),
message.getHand(),
message.getHandState(message.getHand().id()).getPosition());
}
}
});
client.connect();
}
}
A simple 3D demo (Java)
We also provide a simple 3D demo written using the Light Weight Java Game Library (LWJGL) and the provided HandTrackingClient library.
To get a sense of how to use the hand-tracking data, we visualize the 3D hand positions and orientations in a virtual workspace with OpenGL. First, we cache the hand positions and rotations from handleEvent. Note that the x,y,z and w coordinates from the HandTrackingMessage are used to construct a Quaternion.
private Vector3f[] handPositions = new Vector3f[] {new Vector3f(), new Vector3f()};
private Quat4f[] handRotations = new Quat4f[] {new Quat4f(), new Quat4f()};
private boolean[] handPressed = new boolean[] { false, false };
@Override
public synchronized void handleEvent(HandTrackingMessage message) {
// Cache the hand positions, rotations and contact state for rendering
if (baseMessage instanceof PinchMessage) {
PinchMessage message = (PinchMessage) baseMessage;
// Cache the hand positions, rotations and contact state for rendering
if (message.getType() == MessageType.MOVED) {
HandState referencedHand = (message.getHand() == Hand.LEFT) ?
message.getHandState(0) : message.getHandState(1);
handPositions[message.getHand().id()].set(referencedHand.getPosition());
handRotations[message.getHand().id()].set(referencedHand.getRotation());
}
if (message.getType() == MessageType.PRESSED) {
handPressed[message.getHand().id()] = true;
}
if (message.getType() == MessageType.RELEASED) {
handPressed[message.getHand().id()] = false;
}
}
}
Next, we use some helper functions to draw a ground plane and cursors at the hand positions.
/**
* Draw a ground plane and cursors for each hand
*/
private synchronized void render() {
glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glColor3f(1,1,1);
WorkspaceRenderingHelpers.drawGroundPlane(50);
// Show the cursor in yellow if pressing
if (handPressed[0]) glColor3f(1,1,0);
else glColor3f(1,0,0);
WorkspaceRenderingHelpers.drawCursorPoint(handPositions[0], 10);
WorkspaceRenderingHelpers.drawCursorFrame(handPositions[0], handRotations[0], 30);
if (handPressed[1]) glColor3f(1,1,0);
else glColor3f(0,1,0);
WorkspaceRenderingHelpers.drawCursorPoint(handPositions[1], 10);
WorkspaceRenderingHelpers.drawCursorFrame(handPositions[1], handRotations[1], 30);
}
Drawing the skeleton (Java)
Figure 1: Screenshot of the DrawSkeleton example application.
The DrawSkeleton demo demonstrates how to construct and draw the hand skeleton. Hand skeleton information is contained in messages of type POSE. Each pose message contains the coordinate frame of each joint as well as the positions of the finger tips.
Below, we look for messages of type POSE and cast the base message (HandTrackingMessage) to a PoseMessage instance to access the skeleton information. Each PoseMessage structure contains the joint frames and finger tips of both hands. We cache this data for rendering:
@Override
public synchronized void handleEvent(HandTrackingMessage message) {
if (rawMessage.getType() == MessageType.POSE) {
PoseMessage message = (PoseMessage) rawMessage;
for (int iHand=0; iHand<jointFrames.length; iHand++) {
Matrix4f[] jointFrames = message.getJointFrames(iHand);
for (int jJoint=0; jJoint<jointFrames.length; jJoint++) {
this.jointFrames[iHand][jJoint].set(jointFrames[jJoint]);
}
}
for (int iHand=0; iHand<fingerTips.length; iHand++) {
for (int jFinger=0; jFinger<fingerTips[0].length; jFinger++) {
Point3f[] fingerTips = message.getFingerTips(iHand);
this.fingerTips[iHand][jFinger].set(fingerTips[jFinger]);
}
}
}
}
We render the skeleton of each hand by drawing the coordinate frames, the finger tips and connecting the locations of the joint coordinate frames with lines. To draw lines between joints, we need to understand the joint hierarchy of the skeleton:
public static void drawSkeleton(Matrix4f[] jointFrames, Point3f[] fingerTips) {
// Draw the coordinate frame for each joint
for (int i=0; i<jointFrames.length; i++) {
WorkspaceRenderingHelpers.drawCoordinateFrame(jointFrames[i], 5);
}
// Draw the finger tips as points
glColor3f(0,0,1);
glPointSize(5);
glBegin(GL_POINTS);
for (int i=0; i<fingerTips.length; i++)
glVertex3f(fingerTips[i].x,
fingerTips[i].y,
fingerTips[i].z);
glEnd();
// Draw lines connecting the joint locations
//
// T2
// |
// T1 | T3
// \ | /
// J7 J10 J13 T4
// \ | / /
// J6 J9 J12 J16
// T0 \ | / /
// \ \ | / J15
// J4 J5 J8 J11 /
// \ | | | J14
// J3 | | | /
// \ \ | / /
// J2 \ | / /
// \ \ | / /
// \__\|/__/
// J1
// |
// J0
//
// J0: Root
// J1: Wrist
// J2-T0: Thumb
// J5-T1: Index finger
// J8-T2: Middle finger
// J11-T3: Ring finger
// J14-T4: Pinky finger
//
glColor3f(0,1,1);
// Draw a line between the root and the wrist joint
drawLineBetween(jointFrames[0], jointFrames[1]);
// Draw the thumb from wrist to tip
drawLineBetween(jointFrames[1], jointFrames[2]);
drawLineBetween(jointFrames[2], jointFrames[3]);
drawLineBetween(jointFrames[3], jointFrames[4]);
drawLineBetween(jointFrames[4], fingerTips[0]);
// Draw the index finger from wrist to tip
drawLineBetween(jointFrames[1], jointFrames[5]);
drawLineBetween(jointFrames[5], jointFrames[6]);
drawLineBetween(jointFrames[6], jointFrames[7]);
drawLineBetween(jointFrames[7], fingerTips[1]);
// Draw the middle finger from wrist to tip
drawLineBetween(jointFrames[1], jointFrames[8]);
drawLineBetween(jointFrames[8], jointFrames[9]);
drawLineBetween(jointFrames[9], jointFrames[10]);
drawLineBetween(jointFrames[10], fingerTips[2]);
// Draw the ring finger from wrist to tip
drawLineBetween(jointFrames[1], jointFrames[11]);
drawLineBetween(jointFrames[11], jointFrames[12]);
drawLineBetween(jointFrames[12], jointFrames[13]);
drawLineBetween(jointFrames[13], fingerTips[3]);
// Draw the pinky finger from wrist to tip
drawLineBetween(jointFrames[1], jointFrames[14]);
drawLineBetween(jointFrames[14], jointFrames[15]);
drawLineBetween(jointFrames[15], jointFrames[16]);
drawLineBetween(jointFrames[16], fingerTips[4]);
}
Smooth Skinning (Java)
Figure 2: Screen shot of the DrawSkin example application.
The DrawSkin example demonstrates visualizing a 3D model of the hand using smooth skinning and the calibrated hand model. The USER message sent at the beginning of each connection provides the user name and the calibrated, skinned model for each hand. We use linear blend skinning to deform the hand model according to the skeletal pose, given in the POSE messages.
@Override
public synchronized void handleEvent(HandTrackingMessage rawMessage) {
// Cache the skinning information for each hand
if (rawMessage.getType() == MessageType.USER) {
UserMessage message = (UserMessage) rawMessage;
for (int iHand=0; iHand<HandTrackingMessage.N_HANDS; iHand++) {
restPositions[iHand] = message.getRestPositions(iHand);
triangles[iHand] = message.getTriangles(iHand);
skinningIndices[iHand] = message.getSkinningIndices(iHand);
skinningWeights[iHand] = message.getSkinningWeights(iHand);
restJointFrames[iHand] = message.getRestJointFrames(iHand);
skinnedPositions[iHand] = new Point3f[restPositions[iHand].length];
skinnedVertexNormals[iHand] = new Vector3f[restPositions[iHand].length];
for (int i=0; i<skinnedPositions[iHand].length; i++) {
skinnedPositions[iHand][i] = new Point3f();
skinnedVertexNormals[iHand][i] = new Vector3f();
}
}
}
// Deform (i.e. skin) each hand mesh according to the skeletal pose
if (rawMessage.getType() == MessageType.POSE) {
PoseMessage message = (PoseMessage) rawMessage;
for (int iHand=0; iHand<2; iHand++) {
Matrix4f[] jointFrames = message.getJointFrames(iHand);
skin(restPositions[iHand], triangles[iHand], skinningIndices[iHand],
skinningWeights[iHand], restJointFrames[iHand], jointFrames,
skinnedPositions[iHand], skinnedVertexNormals[iHand]);
}
}
}
Other examples, other languages
Both the Echo demo and the Simple3D demo are available in both C++ and Java. In addition, we've included example code for a simple 2D mouse emulator (using java.awt.Robot).
Accessing the depth data
In addition to the network API, we also provide access to the raw depth data to client applications. We use a memory-mapped file to communicate between the hand tracking server and your application. For details, check out our guide to accessing the raw depth data.
Usage tips
This is a preview of the functionality of the hand-tracking API, and we are working on adding support for more gestures. Until then, we recommend keeping your interfaces simple. While it's tempting to use our SDK as a way of simulating the mouse, we recommend designing your interface from scratch for gestures.
A good book on this subject is Daniel Wigdor's Brave NUI World: Designing Natural User Interfaces for Touch and Gesture