Gestures, faces and AR: the traffic officer¶

Exercise gestures-face-ar-annex.pdf · PDF

King Midas asked for a gift: everything he touched would turn to gold. He touched his food. He touched his daughter. A gift you cannot switch off is a curse. Now open the first camera demo and talk to a friend while moving your hands as you normally do. The label jumps: Open_Palm, None, Pointing_Up, Victory. If each of those words ran a command, how many notes have you just deleted?

A camera never stops looking. The hard part of a gesture interface is not recognising the gesture. It is knowing when you did not mean it. This page solves that, then puts a label on your face, and ends with the two questions every camera app must answer: who is watching, and what if I cannot wave?

The officer's signals¶

In 1990, Robert Jacob gave this problem its name, while studying interfaces controlled with the eyes: the Midas touch problem. Everything you do becomes a command.

Think of a traffic police officer at a crossroads. The officer uses few signals, big ones, different from each other, and holds each one until the drivers react. Scratching an ear is not a signal. And the traffic lights are still there: the officer is a second way to direct traffic, not the only one. That is your whole design brief.

MediaPipe Gesture Recognizer returns, for each hand, 21 landmarks and a gesture from a closed list, each with a score from 0 to 1.

The 21 landmarks number the hand from the wrist outwards:

flowchart LR
    W["<b>0</b><br>wrist"] --> T["<b>1 – 4</b><br>thumb<br>(4 = tip)"]
    W --> I["<b>5 – 8</b><br>index<br>(8 = tip)"]
    W --> M["<b>9 – 12</b><br>middle<br>(12 = tip)"]
    W --> R["<b>13 – 16</b><br>ring<br>(16 = tip)"]
    W --> P["<b>17 – 20</b><br>pinky<br>(20 = tip)"]

These are the signals the model knows:

Seven gestures, plus None. If there is no hand at all, the list is simply empty. Can you use "wave goodbye"? No: it is not on the list, and it is a movement, not a hand shape. Design with the signals the officer knows.

Hold to confirm¶

Three protections stand between the camera and your notes:

Here is the referee from pose detection, with a stopwatch:

const vocabulary: Record<string, { label: string; run: () => void }> = {
    Thumb_Up: { label: 'Pin', run: () => box.togglePin() },
    Victory: { label: 'Next note', run: () => box.next() },
    Closed_Fist: { label: 'Delete', run: () => box.remove() },
    Open_Palm: { label: 'Undo', run: () => box.undo() }
};
const HOLD_MS = 800, MIN_SCORE = 0.6;

let candidate = 'None', since = 0, fired = false; // memory between frames
let label = $state(''), progress = $state(0); // what the screen shows

function frame(video: HTMLVideoElement, now: number) {
    const best = recognizer.recognizeForVideo(video, now).gestures[0]?.[0];
    const name = best && best.score >= MIN_SCORE ? best.categoryName : 'None';

    if (name !== candidate) { candidate = name; since = now; fired = false; } // new gesture: restart

    const action = vocabulary[candidate];
    label = action?.label ?? '';
    progress = action ? Math.min(1, (now - since) / HOLD_MS) : 0;

    if (action && progress === 1 && !fired) { action.run(); fired = true; } // once per hold
}

Read it slowly:

• Why are only label and progress $state? Because the user must see that a gesture is being held, and for how long. That is the feedback. The rest is the referee's memory.
• Hold a fist for five seconds. How many notes are deleted? One. To delete another, you have to lower your hand and start again.
• How do you get the note back? Open palm, or the Undo button: forgiveness again.

A longer wait might sound like a slower app. It is not: with a visible progress bar, 800 ms feels deliberate.

Write your vocabulary¶

Before coding, choose 2 to 4 gestures for actions of your own app, and your own HOLD_MS and MIN_SCORE:

The rules:

• A gesture that looks like its meaning is easier to remember: thumb up, "I like it".
• Never give a destructive action to a gesture people make by accident. Open_Palm is every hand at rest.
• The more destructive the action, the longer the hold, and there must always be a way to undo. NoteBox keeps one hold time to stay short, and relies on Undo. Your app can do better.
• Every action is also a button.

Your gesture page¶

This is src/routes/gestures/+page.svelte. You already have CameraStage and createGestureRecognizer from the first demo, and box and NoteList from Voice commands. Anything you put inside <CameraStage>...</CameraStage> is rendered on top of the video, through its children snippet.

<script lang="ts">
    import type { GestureRecognizer } from '@mediapipe/tasks-vision';
    import CameraStage from '#lib/components/CameraStage.svelte';
    import NoteList from '#lib/components/NoteList.svelte';
    import { box } from '#lib/notes.svelte.js';
    import { createGestureRecognizer } from '#lib/nui/vision.js';

    const vocabulary: Record<string, { label: string; run: () => void }> = {
        Thumb_Up: { label: 'Pin', run: () => box.togglePin() },
        Victory: { label: 'Next note', run: () => box.next() },
        Closed_Fist: { label: 'Delete', run: () => box.remove() },
        Open_Palm: { label: 'Undo', run: () => box.undo() }
    };
    const HOLD_MS = 800, MIN_SCORE = 0.6;

    let recognizer: GestureRecognizer;
    let candidate = 'None', since = 0, fired = false; // memory between frames
    let label = $state(''), progress = $state(0); // what the screen shows

    async function load() {
        recognizer = await createGestureRecognizer();
    }

    function frame(video: HTMLVideoElement, now: number) {
        const best = recognizer.recognizeForVideo(video, now).gestures[0]?.[0];
        const name = best && best.score >= MIN_SCORE ? best.categoryName : 'None';

        if (name !== candidate) { candidate = name; since = now; fired = false; } // new gesture: restart

        const action = vocabulary[candidate];
        label = action?.label ?? '';
        progress = action ? Math.min(1, (now - since) / HOLD_MS) : 0;

        if (action && progress === 1 && !fired) { action.run(); fired = true; } // once per hold
    }

    function reset() {
        candidate = 'None';
        label = '';
        progress = 0;
    }
</script>

<h1>Gestures</h1>
<CameraStage onload={load} onframe={frame} onstop={reset}>
    {#if label}
        <div class="hud">
            <span>{label}</span>
            <progress value={progress} max="1" aria-label="Hold to confirm"></progress>
        </div>
    {/if}
</CameraStage>
<p role="status">{label ? `Hold to ${label.toLowerCase()}` : 'Show a gesture to the camera.'}</p>
<NoteList />

<style>
    .hud {
        position: absolute;
        left: 0.75rem;
        bottom: 0.75rem;
        display: grid;
        gap: 0.25rem;
        padding: 0.4rem 0.6rem;
        background: #fff;
        border: 3px solid #111;
    }
</style>

Want more? Add ctx to frame and draw the hand with new DrawingUtils(ctx).drawConnectors(hand, GestureRecognizer.HAND_CONNECTIONS), where hand = result.landmarks[0]. Import both as values, without type.

Detect is not recognise¶

These three look similar. They are very different things, and one of them comes with legal weight:

Face detection only finds a face-shaped region. The same box would appear for anyone in the room, so it knows nothing about who you are. If your app needs to know who, it asks for a login, not for a face.

Biometric data, in the GDPR (article 4.14), is personal data from a specific technical processing of a person's physical, physiological or behavioural characteristics, which allows or confirms their unique identification. Facial images and fingerprints are its examples.

Glue on the real world¶

Augmented reality (AR) adds virtual elements to the real world, live, and places them where they belong.

In 1997, Ronald Azuma defined it with three conditions. It combines real and virtual. It is interactive in real time. And the virtual part is registered with the real world: it stays glued to its place.

A film with special effects fails the second condition. A logo in the corner of a video fails the third. A label that floats above your head and follows you passes all three.

The formula¶

The detector gives you a box in video pixels: originX, originY, width, height. (Careful: the 6 key points of a detection come normalised from 0 to 1, as landmarks do. The box does not.) Your sticker is an HTML element with position: absolute, sized in percentages. Say the video is 640 wide and the face is at originX = 200, with width = 160:

Why 100 −? The video is shown mirrored. A face on the right of the camera picture is on the left of the screen, and the box's right edge (originX + width) becomes the sticker's left edge.

Why percentages? The video can be any size on the screen, and percentages follow it. In pixels it would work at one window size and break at another.

A sticker on your face¶

Add the detector to vision.ts, and FaceDetector to the import at the top:

export async function createFaceDetector() {
    return FaceDetector.createFromOptions(await vision(), {
        baseOptions: {
            modelAssetPath: `${MODELS}/face_detector/blaze_face_short_range/float16/latest/blaze_face_short_range.tflite`,
            delegate: 'GPU'
        },
        runningMode: 'VIDEO'
    });
}

Then the page, src/routes/ar/+page.svelte. The sticker shows the current note. And a face triggers an action: with two faces in the picture, somebody may be reading over your shoulder, so the note blurs.

<script lang="ts">
    import type { FaceDetector } from '@mediapipe/tasks-vision';
    import CameraStage from '#lib/components/CameraStage.svelte';
    import { box } from '#lib/notes.svelte.js';
    import { createFaceDetector } from '#lib/nui/vision.js';

    type Sticker = { left: number; top: number; width: number }; // percentages of the video

    let detector: FaceDetector;
    let stickers = $state<Sticker[]>([]);

    async function load() {
        detector = await createFaceDetector();
    }

    function frame(video: HTMLVideoElement, now: number) {
        const result = detector.detectForVideo(video, now);
        stickers = result.detections.flatMap((detection) => {
            const face = detection.boundingBox; // video pixels, origin top-left
            if (!face) return [];
            const width = (face.width / video.videoWidth) * 100;
            const left = (face.originX / video.videoWidth) * 100;
            return { left: 100 - left - width, top: (face.originY / video.videoHeight) * 100, width };
        });
    }
</script>

<h1>AR</h1>
<CameraStage onload={load} onframe={frame} onstop={() => (stickers = [])}>
    {#each stickers as sticker, i (i)}
        <div class="sticker" style:left="{sticker.left}%" style:top="{sticker.top}%" style:width="{sticker.width}%">
            {box.current?.title ?? 'NoteBox'}
        </div>
    {/each}
</CameraStage>

<article class:blurred={stickers.length > 1}>
    <h2>{box.current?.title}</h2>
    <p>{box.current?.body}</p>
</article>
{#if stickers.length > 1}
    <p role="status">Two faces detected: the note is hidden.</p>
{/if}

<style>
    .sticker {
        position: absolute;
        transform: translateY(-110%);
        box-sizing: border-box;
        padding: 0.2rem 0.4rem;
        background: #ffd60a;
        border: 3px solid #111;
        text-align: center;
    }
    .blurred {
        filter: blur(8px);
    }
</style>

Treat the blur as a courtesy, not as security. A face on a poster may count as a second face, and the short-range model is made for selfie distance, so somebody far behind you may not count at all.

WebXR, the real thing¶

Your sticker is glued in 2D, on a picture. To put a sofa on your real floor, at real size, the device has to understand the room in 3D.

The WebXR Device API is the web standard for AR and VR. The browser gives the page the position of the device and the surfaces it sees. An AR session has the mode immersive-ar.

Where does it work? (Checked against MDN's compatibility data, November 2026. MDN marks WebXR as experimental, with limited availability.)

WebXR also needs a secure context, like the camera. So you ask before you offer. One call tells you:

// npm install -D @types/webxr   +   /// <reference types="webxr" />  in src/app.d.ts
const supported = await (navigator.xr?.isSessionSupported('immersive-ar') ?? Promise.resolve(false)).catch(() => false);

navigator.xr does not exist where there is no WebXR, and the promise can be rejected, for example when a permissions policy blocks it. In both cases, the answer for your interface is the same: no.

If it is false, do not show an "Enter AR" button that does nothing. Show the video overlay instead. That is your fallback. Starting a real AR session also needs a 3D engine such as three.js or A-Frame, which is out of scope for this course. And no, AR does not need a headset: a phone is enough for WebXR, and a webcam is enough for an overlay.

Two questions¶

Every camera or microphone app must answer two questions. Use this list to audit your own app, then swap with a classmate and audit theirs:

Hiding the video does not turn the camera off. Only stopping every track does. And the promises on your page have to be true: if you say "never uploaded", your code must never upload.

These have official names in WCAG 2.2:

So 2.5.4 asks for two things, and the second is easy to forget. The camera being off by default, behind a button, is how your app meets it: turn the camera off, and the motion is ignored.

Key takeaways¶

• Gesture Recognizer returns 21 landmarks and one of 7 gestures (or None) with a score. Few, big, different signals: the traffic officer.
• The Midas touch problem is that everything you do becomes a command. Defuse it with a minimum score, a hold time with visible progress, and one action per hold.
• Face detection says where a face is; face recognition says whose. Biometric data used to identify someone is a special category under GDPR article 9. Our apps detect, never identify, and never store or send a frame.
• Augmented reality combines real and virtual, in real time, registered to the world. For a mirrored video: left = 100 − (originX + width) / videoWidth × 100, in percentages.
• WebXR immersive-ar needs AR hardware: an Android phone with ARCore or a Meta Quest. Not laptops, not Firefox, not Safari on iPhone. Ask with isSessionSupported, and keep a fallback.
• Audit with two questions: who is watching? and what if I cannot? WCAG 2.5.4 Motion Actuation requires a non-motion way in, and a way to switch the motion off.

Quick check¶

You have given your app senses and asked the two questions every sensor has to answer. In the next unit, the data comes back out, in a form people can read and print: Report structure.