How hand tracking works in the browser: A Complete Guide

You can try it now. Open Ink, turn on the camera and point with your index finger to draw a thread of colour. Bring your thumb and index finger together to gather a drop, then separate them to let it bloom in the water.

This guide follows that pipeline step by step. It covers what the camera permission actually grants, what a model like MediaPipe Hand Landmarker returns, how a few dozen numbers become a gesture, what stays on your device, and where the technique still struggles. If you want the wider question of what makes art interactive or generative, Interactive generative art, explained covers that ground. Here the focus is on the mechanism.

How it works, in five steps

The page asks for the camera. The browser shows a permission prompt, and nothing is captured until you accept.
Frames go to a model. Each video frame is handed to a vision model running inside the page.
The model returns landmarks. For each hand it finds, it outputs 21 points: the wrist and the joints and tips of the fingers.
Code derives gestures. Distances, angles and speeds between those points become states such as pointing, pinching, an open palm or a fist.
Gestures become parameters. Those states are mapped onto a visual or sound system, such as a position, a force or a colour.

The rest of this guide takes each step in turn.

Step 1: Asking for the camera

A web page cannot see through your webcam by default. To get video, it calls a browser function named getUserMedia(), part of the Media Capture and Streams specification published by the W3C. MDN's reference for getUserMedia describes what happens next. The browser asks the user for permission, and only if it is granted does the page receive a MediaStream, a live stream of frames. If the user refuses, or there is no camera, the request fails and the page receives nothing.

Some details of this step matter for privacy and for how the experience feels:

The request is specific. A page can ask for video only, audio only, or both, and can say which camera it prefers (for example, the one facing the user). A hand-tracking page has no reason to ask for the microphone.
It needs a secure context. According to MDN, getUserMedia() is only available on pages served over HTTPS (or from a local development address). An ordinary insecure page cannot request the camera at all.
The browser stays in charge. Browsers show an indicator while the camera is active, and you can revoke the permission from the site settings at any time. The rules for this are defined in the W3C Media Capture and Streams specification.

Note what this step does not involve. Granting camera access does not mean the video is shown on screen, recorded or sent anywhere. The page simply receives frames in memory. What it does with them is a separate question, and the privacy section below returns to it.

Step 2: From frames to a model

Once the stream is running, the page reads frames from it, typically by attaching the stream to a hidden <video> element and sampling it on every animation frame. Each sampled frame is an image, a grid of pixels, and it is passed to a model.

The most widely used model for this job on the web is MediaPipe Hand Landmarker, from Google's MediaPipe project. Its documentation describes a task that "operates on image data with a machine learning (ML) model as static data or a continuous stream." It ships with guides for Android, Python and the web. On the web, the model file is downloaded by the page, and the computation runs inside the browser through a WebAssembly runtime, with optional GPU acceleration.

The task has three running modes. IMAGE handles single pictures and VIDEO handles decoded video files. LIVE_STREAM is the one for a webcam, and in that mode results arrive through a listener as frames are processed. Several settings tune its behaviour. num_hands sets how many hands to look for, and the default is 1. Three confidence thresholds (detection, presence and tracking, each 0.5 by default) decide how sure the model must be before it reports a hand or keeps following one.

Two models do the work, packaged together in one bundle. According to Google's documentation:

A palm detection model looks across the whole frame for something that looks like a hand.
A hand landmarks detection model then looks closely at that region and places the points.

This split is why tracking can stay light. In video and live-stream modes, once a hand has been found, the landmark model can follow it from frame to frame. The heavier palm detection only needs to run again when the tracker loses confidence that a hand is present.

Step 3: Twenty-one landmarks

The output that matters is the set of landmarks. MediaPipe's hand model returns 21 points per hand, which Google's documentation calls "21 hand-knuckle coordinates":

1 point for the wrist
4 points for each of the five fingers: the base joint, two middle joints and the tip

Each point comes in two forms. Image coordinates are x and y, normalised to the width and height of the frame, plus a relative depth value. World coordinates are rough 3D positions in metres, with the origin at the hand's approximate geometric centre. The model also reports handedness, a guess at whether each hand is a left or a right.

It helps to picture the result as a small skeleton made of dots, redrawn many times per second over a video the page never needs to show you. That skeleton is the only thing the rest of the system works with. From here on, the code is no longer dealing with an image of a hand. It is dealing with 21 points and their movement.

The documentation also says the model was trained on about 30,000 real-world images, plus synthetic hand models rendered over various backgrounds. That partly explains why it copes with many skin tones, rooms and lighting conditions, and also why it can still be fooled by unusual angles.

Step 4: From points to gestures

Twenty-one moving points are not yet a gesture. Turning them into something a person can intend, and repeat, is the craft of hand-tracking interfaces. The model does not decide that you are pinching. The application's code decides, from geometry.

Some common derivations:

Pointing and position. The tip of the index finger (or the centre of the palm) becomes a cursor. Its x and y map onto the screen, usually mirrored so that moving your hand right moves the cursor right, as in a mirror.
Pinch. The distance between the thumb tip and the index tip is measured. When it falls below a threshold, relative to the size of the hand so it works near and far from the camera, a pinch begins. When it rises above a slightly larger threshold, the pinch ends. The gap between the two thresholds stops the state from flickering.
Open palm and fist. How far the fingertips sit from the wrist or palm centre, compared with their joints, says whether the fingers are extended or curled. Averaged over the fingers, this gives a continuous value between open and closed, and not only a yes or no.
Speed and direction. Comparing a point's position across frames gives velocity, so a quick horizontal sweep can be told apart from a slow drift.

Two practical problems come up at once. The first is jitter: landmarks shake slightly even when your hand is still, so most applications smooth them over a few frames and accept a little delay in exchange for calm movement. Synfield does this with a One Euro filter on the tracked point, a speed-adaptive low-pass that keeps still hands calm without making quick moves feel sticky. The second is reach: the area your hand can comfortably cover is smaller than the camera's view, so Synfield lets you calibrate the edges of that area and stores them locally as a "hand area" preference (gestureos.calibration.v1 on the privacy page).

Step 5: From gestures to a living system

The last step is where hand tracking stops being a technical demo and becomes an experience. The gesture states are mapped onto parameters of whatever the page is running, such as a particle field, a fluid, a sound or a set of reflections.

The mapping is a design decision, and the same small vocabulary of gestures can mean very different things. In Synfield's worlds, for example:

In Helix, pointing winds a current of light, a held pinch draws it in and breaks it into spray, and releasing lets a wave of light form again. Opening or closing your hand lays the wave out or curls its crest.
In Mirror, your index finger draws a stroke that is reflected around a centre. A held pinch moves that centre, and an open palm or a fist adds or removes folds.
In Breathe, the hand barely commands anything. A pinch pauses the breathing light and releasing lets it continue, while opening or closing your hand widens or softens it.

The pipeline is identical in all three. Only the mapping changes. That is the useful lesson for anyone building with hand tracking: the model gives you the same 21 points every time, and the character of the work depends on what you let those points mean. For how such mappings fit into generative systems more broadly, see the camera section of Interactive generative art, explained.

Is browser hand tracking private?

It can be, but it depends on how a particular site is built. The technique does not guarantee it. Be precise about which parts are fixed by the browser and which parts are choices made by the site.

What the browser guarantees. Under the Media Capture specification and browser practice described by MDN, a page cannot read the camera without your permission, must be served securely to ask, and shows an indicator while the camera is in use. You can withdraw permission at any time.

What depends on the site. Once a page has frames, its code can do anything with them. That includes processing them locally, drawing them on screen, recording them, or uploading them to a server. MediaPipe Hand Landmarker is designed to run on the device, inside the browser, so a site using it does not need to send video anywhere to track your hand. Whether it does is up to the site, so the honest test is to read its privacy page and, if you can, watch its network activity.

What Synfield says it does. Synfield's privacy page describes the site as it is built today:

The camera is off when a world opens. It starts only if you choose "Explore with your hands", or turn it on later from the camera control or the C key.
The request is for video from the camera facing you, with no microphone, and the picture is not shown on the page.
Frames are read in the page and passed to a vision task that also runs in the browser. It looks for one hand and returns points. The model is a file on the site itself and downloads only after the camera starts.
The site's code does not record the video, does not upload the frames or the hand points, and does not create an account.
The anonymous visit counter records only coarse facts, such as whether a mouse, touch or hands were used and when the camera was turned on. Nothing from the camera image or the hand points is included.

Those statements match the shipped code: Synfield loads MediaPipe Hand Landmarker (@mediapipe/tasks-vision) with the site's own hand_landmarker.task model file, sets numHands to 1, and tries a GPU delegate first with a CPU fallback.

Choosing "Hands" on a world's information page only shows instructions and never turns on the camera. The camera starts only inside a world, and only when you ask.

The stack, briefly

You do not need to know the tooling to enjoy a hand-tracked artwork, but it helps to know what the names mean.

getUserMedia / Media Capture and Streams. This is the browser standard for camera and microphone access. Every webcam-based experience on the web starts here. (MDN, W3C)
MediaPipe Hand Landmarker. Google's current task for hand landmarks, with a web package that loads a model bundle and runs it in the page. It returns landmarks, world landmarks and handedness. (Google AI Edge documentation)
TensorFlow.js and earlier hand models. Before the current MediaPipe Tasks API, browser hand tracking was commonly done with TensorFlow.js models such as Handpose and with the older "MediaPipe Hands" solution. Many tutorials still use those names, so if you see them, they describe an earlier generation of the same idea.
A rendering layer. The landmarks then drive whatever draws the picture, usually WebGL or Canvas. That part is ordinary graphics programming, and it is where most of the artistic decisions are made.

Limits, and why mouse and touch still matter

Hand tracking in the browser works well enough to make art with. It does not work everywhere, and it is not the best input for everyone. The common limits:

Light. The model sees pixels. A dim room, a strong light behind you or a hand in shadow all lower its confidence, and it may lose the hand.
Distance and framing. Too close and parts of the hand leave the frame. Too far and the hand is only a few pixels wide. An arm's length from a laptop camera is usually comfortable.
Occlusion. When fingers overlap or the hand turns edge-on to the camera, the model has to guess where hidden joints are, and gestures such as a pinch become less reliable.
Two hands and other people. More hands mean more ambiguity and more computation. Many experiences track only one hand for this reason.
Devices. Inference needs processing power. Recent laptops handle it comfortably. Older or low-power devices may track at a lower rate or warm up noticeably, and mobile browsers vary. Synfield does not publish a closed allow-list: hand mode needs a secure context, getUserMedia, and a browser that can run the vision WASM (with GPU when available). Recent desktop browsers are the comfortable path; on phones the same stack can work, but framing and heat vary, so touch is usually easier.
Bodies. Not everyone can hold an arm up for long, and not everyone wants to be on camera. An interaction that only works with hands leaves people out.

For that last reason, the most careful hand-tracked work treats the camera as one way in among several. In Synfield, every world can also be explored with a mouse or a touch screen, and the gestures have direct equivalents. Holding the mouse button or resting a finger does what a pinch does, and the wheel or a two-finger spread does what opening and closing the hand does. The How to explore guide lists them side by side.

A short note on history

The idea of a body steering an image without touching anything is older than webcams. Myron Krueger's Videoplace, from the 1970s and 1980s, used cameras to put a person's silhouette into a responsive artwork. That history belongs to the story of interactive art, told in What is interactive art?. What is new is the setting. Detecting a detailed hand once needed special hardware, and now it is a model file a web page can download and run in a tab.

Try it

The best way to understand the pipeline is to feel it respond. Start with one of these, then wander:

Ink: pinch to gather a drop, release to let it bloom.
Helix: point to wind a current of light.
Mirror: draw one stroke and watch it fold.
Breathe: a slower world, where a pinch only pauses the light.
Matter: an open palm repels a particle field and a fist attracts it.

Or open Explore and choose "Explore with your hands". The camera stays off until you do.

FAQ

Do I need to install anything for hand tracking in the browser? No. A modern browser and a webcam are enough. The page downloads the model and runs it itself. You only grant camera permission.

Is webcam hand tracking private? The browser guarantees that a page cannot use your camera without permission. Whether frames stay on your device depends on the site. Models like MediaPipe Hand Landmarker run locally, so uploading is not needed. Check the site's privacy page. Synfield's says frames and hand points are not recorded or uploaded.

What are the 21 hand landmarks? They are the points MediaPipe's model places on each detected hand: one for the wrist and four for each finger, from the base joint to the tip. Gestures are calculated from the distances and angles between them.

Does it work on a phone? Often, yes, where the mobile browser supports camera access and can run the on-page vision model. Performance and framing vary by device, so touch is usually the more comfortable way in on a phone.

What can I do if hand tracking loses my hand? Add light in front of you, move about an arm's length from the camera, and keep your palm facing it. If it still struggles, a mouse or touch screen gives the same experience.