Leeds Hand Tracker Demo

hits since 21st May 1996

New Version 2.1 (29/10/96) features improved mouse controller - button clicks are now possible!!!!!!

Overview

The Leeds Hand Tracker was developed by Tony Heap. An unmarked human hand can be tracked from a single colour video camera connected to a suitable workstation. An uncluttered background is not necessary and the tracker can be trained specifically to one hand colour to improve performance. The current version of the tracker is limited to being able to track a hand held palm out with all fingers showing. We are currently developing a more sophisticated version in which full unrestricted movement is possible.

At the heart of the system is a deformable outline model of the hand, which has been constructed from a number of examples of hands in different positions. The hand is tracked from frame to frame using a non-trivial edge detection technique to drive the model towards the hand in the image. Full technical details can be found in Olivetti Research Limited Tech Report 95.1. Here's a snapshot of the tracker in action...

...and you can click here for a video clip (100Kb MPEG).

Try it out...

If you have a Silicon Graphics Indy(TM) running Irix 5.3 or later, you can try the hand tracker out for yourself.

Downloading

You can download HandTrackerV2.1.tar.Z onto your host machine (350Kb). To unarchive it, type:

uncompress HandTrackerV2.1.tar.Z
tar -xf HandTrackerV2.1.tar

This generates the handtracker directory with everything you need in it.

Instructions for use

Before running the demo, you should check that the IndyCam is set up properly. Type vcp to load the video control panel, and check that the default input is `IndyCam' and not `Analog Source'. Select `Live Video Input' from the `Utilities' menu to check that a signal is coming through. Quit the live video input before running the tracker demo. To run the tracker demo, type this:

cd handtracker/demo
tracker

This should bring up a window with the view from the IndyCam in it (reversed so it acts like a mirror). There is a hand template in the middle of the view. If you move your right hand up to the template it should `lock on' to your hand and follow it around from there. The tracker can only track your hand so long as the palm and all fingers are showing. If it thinks it has lost your hand it resets to the central position. If it's stuck on the wrong part of your hand, remove your hand from view and it will reset.

Filename Problems

If you get an error like load_linear_mean: Can't open "../models/hand-lin.mean" for reading, then the tracker can't find the hand model - you need to use the -m switch to tell it where the model files (hand-lin.*) are, eg.

tracker -m ./hand-lin L

The L is necessary, but don't worry about its meaning!

Lighting conditions

The tracker's performance can be improved by adjusting the colour filtering to suit your particular lighting conditions. To check the colour filtering, run the tracker in training mode:

tracker -train

Along with the normal view there will be a smaller filter window, showing the image after colour filtering. It should look something like this:

There is good definition of the hand and it should track well. If it looks like either of these:

then it might be worth altering the colour filtering. The tracker has an auto-train facility for this. Hold your hand up to the IndyCam so that the whole of the small square is filled with hand-coloured pixels. Then (with your other hand) click the left mouse button. The pixels are sampled and a colour histogram is built. The new filter is then used in the filter window. You can take more samples by repeating this process. To clear the histogram and start again, click the middle button. When (if) you're happy, click the right mouse button to save the histogram (the filename used is tracker.hist).

To use your trained colour filter, type:

tracker -hist <histogram filename>

Performance of the auto-trainer is sometimes marred by poor lighting or by the relatively low quality of the IndyCam. If you can't get good results by auto-training, you can manually alter the red/green/blue components of the colour filter to make it respond bettor to your skin tones. Try:

tracker -train -f <r> <g> <b>

Where R, G and B are the three colour filter values (used additively, so the filtered greyscale value for a pixel with colour components r, g and b is r*R + g*G + b*B). The defaults are 4.0, -2.0 and -2.0 respectively which has the effect of highlighting red but damping white.

If you're still having problems there are two other measures. Firstly, find a source of ambient light (such as an anglepoise lamp) and position it to illuminate your hand. Secondly, go to a hardware store and buy some bright yellow rubber gloves. Then use the auto-trainer to respond to them. If this doesn't work, I don't know what will!

Command line options

When you run the tracker you can alter its behaviour with one or more command line switches:

-m <model base filename> <model type L/P/H> is to specify the hand model filename. The only model available at the moment is hand-lin so there ain't much choice, but you can include the full or relative pathname if yours is different to ../models/hand-lin.

-v <number of variation modes> specifies the number of degrees of freedom in the variation of the mand model. The default is 4. Decrease to 3, 2, 1 or 0 for slight speed increase coupled with less robustness.

-i <number of iterations> is the number of tracking iterations per frame. There is a speed/performance tradeoff. Default is 4.

-step <value> alters the tracker sensitivity ie. how far the model can move per iteration. There is a tradeoff between speed and stability. Default is 1.0.

-rotate alters the initial orientation of the hand template. -90 degrees is upright.

-lefthand reverses the hand template for left handed use.

-overhead configures the tracker for use with an overhead camera pointing down onto a desktop.

-mouse puts the tracker in mouse pointer control mode. See below for details.

-outputpos outputs hand position info to stdout so you can pipe it into your own applications. See below for more details.

-train runs the tracker in `lighting condition training' mode. See `lighting conditions' below for details.

-hist <filename> specifies the colour histogram filename. See `lighting conditions' below for details.

-f <red filter value> <green filter> <blue filter> sets the colour filter values to suit your lighting conditions. See `lighting conditions' below for details. Defaults are 4.0, -2.0, -2.0.

-w <size> specifies the relative window size. Default is 1, other valid options are 2 and 0.5. Using a smaller window speeds up the tracker a bit since there are less pixels to write to the screen. Using a larger windows slows it down a lot.

-d <interval> dictates how frequently the IndyCam view is echoed to screen. Default is 1; increasing to 2 displays every other frame, 3 does every 3rd frame etc. This is purely to speed the tracker up by way of fewer screen-writes.

-r periodically outputs the frame rate.

-s outputs the number of model points which have `found an edge' in each frame.

-thicklines draws the hand outline thicker so it's more visible.

-noannotate suppresses the sizing circle and angle line which usually accompany the hand outline.

Using the hand tracker to control the mouse pointer

Yes! You can now control the mouse pointer without even touching the mouse! Try:

tracker -mouse

The pointer moves with the position of your hand. You perform mouse button clicks by moving your thumb.

Using the hand tracker to control your own applications

You can use the hand tracker to control your own applications if you want. There are two ways to do this. The simplest is to use the tracker in output mode by typing:

tracker -outputpos

The tracker spurts position info to stdout which can be piped into you application or whatever. The format of the output is as follows:

X = <x coordinate of centre of hand>
Y = <y coordinate of centre of hand>
SIZE = <approximate width of hand in pixels, divided by two>
ANGLE = <angle of hand to horizontal in degrees (upright is -90 deg)>
DEFORM = <list of deformation parameters from which hand shape can be
	  determined (default is 4 params, but can be overridden by -v
	  switch)>
THUMB_IN = <0 if the thumb is out, 1 if the thumb is in>

I plan to include other parameters in a future version, such as the position of the thumb (`in' or `out'), so allow for unrecognized parameters in any code you write. Also, let me know if you have any special requests.

A more sophisticated way to link the hand tracker to your own applications has been developed by Richard Bowden; it's geared primarily towards (but not limited to) Inventor(TM) applications. Click here for more details.

Tony Heap - ajh@comp.leeds.ac.uk