Natural user interfaces: the app gets senses¶

Exercise natural-user-interfaces-annex.pdf · PDF

Put both hands in your pockets. Could you still unlock your phone? Probably: it looks at your face. Could you set a timer? "Hey Siri" does that. Have you ever played a game with your whole body? No mouse, no keyboard, and the machine still understood you.

So far your apps have only felt one fingertip at a time: a click, a key. In this unit your app gets senses: an ear, a mouth and eyes. By the end of this page, a page of your own SvelteKit project will see your hand through the webcam, with about twenty lines of your code.

CLI, GUI, NUI¶

You met the first two in Apple's guidelines. With a command-line interface (CLI) you remember a command and type it. With a graphical user interface (GUI) you see the options and point at one. The third step removes the pointer too.

A natural user interface (NUI) is an interface you operate with what you already do as a person: speaking, moving your hands, moving your body, looking around.

It took decades to get from a lab to your pocket:

Four families¶

Every NUI product belongs to at least one of four families. Ask two questions: which sensor does it use, and what does the machine perceive?

Some products sit in two families. An app that lets you try on glasses has to find your face (hands and face), then draws the glasses on it (AR). And some need no intelligence at all: a pinch is just two fingers on a touch screen.

Senses are not understanding¶

Your ear only catches air vibrating. A part of your brain turns that into words. Then you decide what to do about them. If a friend says "jump", your ear and your brain have worked perfectly, and you may still decide not to jump.

Every NUI has the same three parts, plus a fourth: it answers.

flowchart LR
    S["<b>Sensor</b><br>webcam picture<br>(the eye)"] -->|"numbers"| M["<b>Model</b><br>Thumb_Up, 93 %<br>(the brain recognising)"]
    M -->|"a label and a score"| R["<b>Rule</b><br>your code decides<br>(you deciding)"]
    R -->|"what happened"| F["<b>Feedback</b><br>the note shows pinned<br>(you answering)"]

A machine learning model is a program that nobody wrote rule by rule. It was trained with thousands of examples until it learnt to give the right answer by itself.

The toolbox¶

You do not have to train anything. These tools already exist for web interfaces:

Careful with one common mix-up: MediaPipe is not "the AI of the browser". It is a library you install. What the browser itself gives you is the Web Speech API and WebXR.

Under the hood, the libraries lean on two engines of the browser: WebAssembly (a compact format that runs fast code on the processor) and WebGL or WebGPU (ways of using the graphics card).

Where does the picture go?¶

When you show your hand to a MediaPipe demo, where does the picture go? Most people guess "to Google". It does not.

MediaPipe runs in the browser. Your computer does the work, inside the tab, frame by frame. A lot of voice assistants do the opposite.

That is why this approach suits sensitive images, and why it does not suit everything. The models in this unit weigh less than 10 MB. A large chatbot model weighs gigabytes.

Your first demo¶

Now the eye opens. You need Chrome or Edge 139 or newer, a webcam and your own SvelteKit project, the one you built in Unit 3 and Unit 4.

1. Install the library, at an exact version.

npm install --save-exact @mediapipe/tasks-vision@1.0.1

Why exact? The library in node_modules and the WebAssembly runtime it downloads from the CDN must have the same version, or you get errors in the console. A plain npm install gives you whatever is newest, while the URL in vision.ts stays at 1.0.1. Never write @latest in that URL.

2. Add three ready-made files. They do the boring part: opening the camera, loading the model, keeping the camera's states. Read them at home: there is nothing here you have not seen.

// Opening the webcam, and turning every way it can fail into one word the UI can show.
export type CameraProblem = 'unsupported' | 'denied' | 'no-camera' | 'busy' | 'unknown';

export async function openCamera(): Promise<MediaStream> {
    // mediaDevices only exists in a secure context: https:// or http://localhost
    if (!navigator.mediaDevices?.getUserMedia) throw new DOMException('No camera API', 'NotSupportedError');
    return navigator.mediaDevices.getUserMedia({
        video: { width: 640, height: 480, facingMode: 'user' },
        audio: false
    });
}

export function cameraProblem(error: unknown): CameraProblem {
    const name = error instanceof DOMException ? error.name : '';
    if (name === 'NotAllowedError' || name === 'SecurityError') return 'denied';
    if (name === 'NotFoundError' || name === 'OverconstrainedError') return 'no-camera';
    if (name === 'NotReadableError' || name === 'AbortError') return 'busy';
    if (name === 'NotSupportedError') return 'unsupported';
    return 'unknown';
}

export const cameraProblemText: Record<CameraProblem, string> = {
    unsupported: 'This page cannot use the camera. Open it over https or on localhost.',
    denied: 'Camera blocked. Allow it from the icon in the address bar, then press Try again.',
    'no-camera': 'No camera found. Plug one in, or keep using the buttons.',
    busy: 'The camera is in use by another app. Close it, then press Try again.',
    unknown: 'The camera could not start. Keep using the buttons.'
};

// MediaPipe Tasks for web: load the WebAssembly runtime once, then create the task you need.
import { FilesetResolver, GestureRecognizer } from '@mediapipe/tasks-vision';

// Pin the same version as package.json: runtime and library must match.
const WASM = 'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@1.0.1/wasm';
const MODELS = 'https://storage.googleapis.com/mediapipe-models';

let fileset: ReturnType<typeof FilesetResolver.forVisionTasks> | undefined;
const vision = () => (fileset ??= FilesetResolver.forVisionTasks(WASM));

export async function createGestureRecognizer() {
    return GestureRecognizer.createFromOptions(await vision(), {
        baseOptions: {
            modelAssetPath: `${MODELS}/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task`,
            delegate: 'GPU'
        },
        runningMode: 'VIDEO',
        numHands: 1
    });
}

<script lang="ts">
    import type { Snippet } from 'svelte';
    import { cameraProblem, cameraProblemText, openCamera, type CameraProblem } from '#lib/nui/camera.js';

    type Status = 'off' | 'loading' | 'on' | CameraProblem;

    type Props = {
        /** Load the model here. Called once, the first time the camera is turned on. */
        onload?: () => Promise<void>;
        /** Called for every new video frame. `now` is a timestamp in milliseconds. */
        onframe: (video: HTMLVideoElement, now: number, ctx: CanvasRenderingContext2D) => void;
        /** Called when the camera stops, so the parent can reset what it shows. */
        onstop?: () => void;
        /** HTML drawn on top of the video (labels, AR stickers). */
        children?: Snippet;
    };
    let { onload, onframe, onstop, children }: Props = $props();

    let status = $state<Status>('off');
    let video: HTMLVideoElement;
    let canvas: HTMLCanvasElement;
    let stream: MediaStream | null = null;
    let frame = 0;
    let loaded = false;
    let lastTime = -1;

    async function start() {
        status = 'loading';
        try {
            stream = await openCamera(); // 1. permission prompt
            if (!loaded) {
                await onload?.(); // 2. download the model (a few MB, cached afterwards)
                loaded = true;
            }
            video.srcObject = stream;
            await video.play();
            canvas.width = video.videoWidth;
            canvas.height = video.videoHeight;
            status = 'on';
            frame = requestAnimationFrame(loop); // 3. one detection per frame
        } catch (error) {
            stop();
            status = cameraProblem(error);
        }
    }

    function loop() {
        const ctx = canvas.getContext('2d');
        // Only run the model when the camera has delivered a new frame.
        if (ctx && video.currentTime !== lastTime) {
            lastTime = video.currentTime;
            ctx.clearRect(0, 0, canvas.width, canvas.height);
            onframe(video, performance.now(), ctx);
        }
        frame = requestAnimationFrame(loop);
    }

    function stop() {
        cancelAnimationFrame(frame);
        stream?.getTracks().forEach((track) => track.stop()); // the camera light goes off
        stream = null;
        if (video) video.srcObject = null;
        status = 'off';
        onstop?.();
    }

    // Leaving the page must switch the camera off.
    $effect(() => stop);
</script>

<section class="stage">
    <div class="view" class:hidden={status !== 'on'}>
        <!-- muted + playsinline: required for autoplay on phones -->
        <video bind:this={video} muted playsinline aria-label="Your webcam, mirrored"></video>
        <canvas bind:this={canvas} aria-hidden="true"></canvas>
        {@render children?.()}
    </div>

    <div class="bar" role="status" aria-live="polite">
        {#if status === 'off'}
            <p>Camera off. The video is analysed on this computer and never uploaded.</p>
            <button type="button" onclick={start}>Turn camera on</button>
        {:else if status === 'loading'}
            <p>Starting the camera and loading the model...</p>
        {:else if status === 'on'}
            <p><span class="dot" aria-hidden="true"></span> Camera on</p>
            <button type="button" onclick={stop}>Turn camera off</button>
        {:else}
            <p class="problem">{cameraProblemText[status]}</p>
            <button type="button" onclick={start}>Try again</button>
        {/if}
    </div>
</section>

<style>
    .stage {
        border: 3px solid #111;
        max-width: 640px;
    }
    .view {
        position: relative;
        aspect-ratio: 4 / 3;
        background: #111;
        overflow: hidden;
    }
    .view.hidden {
        display: none;
    }
    video,
    canvas {
        position: absolute;
        inset: 0;
        width: 100%;
        height: 100%;
        object-fit: cover;
        transform: scaleX(-1); /* mirror, like a selfie camera */
    }
    .bar {
        display: flex;
        gap: 0.75rem;
        align-items: center;
        justify-content: space-between;
        flex-wrap: wrap;
        padding: 0.6rem 0.8rem;
    }
    .bar p {
        margin: 0;
    }
    .problem {
        border-left: 6px solid #d62828;
        padding-left: 0.5rem;
    }
    .dot {
        display: inline-block;
        width: 0.7rem;
        height: 0.7rem;
        border-radius: 50%;
        background: #d62828;
    }
    button {
        font: inherit;
        padding: 0.4rem 0.8rem;
        border: 3px solid #111;
        background: #fff;
        cursor: pointer;
    }
</style>

CameraStage is a component like the ones you built in Unit 4: props in, callbacks out. You give it two functions. onload runs once, to download the model. onframe runs for every new video frame.

3. Create the page. This is src/routes/demo/+page.svelte. Type it, do not paste it: you will write this loop again on every page of this unit.

<script lang="ts">
    import type { GestureRecognizer } from '@mediapipe/tasks-vision';
    import CameraStage from '#lib/components/CameraStage.svelte';
    import { createGestureRecognizer } from '#lib/nui/vision.js';

    let recognizer: GestureRecognizer;
    let gesture = $state('None');
    let score = $state(0);

    async function load() {
        // once: download the model
        recognizer = await createGestureRecognizer();
    }

    function frame(video: HTMLVideoElement, now: number) {
        // for every video frame
        const result = recognizer.recognizeForVideo(video, now);
        const best = result.gestures[0]?.[0]; // first hand, most likely gesture
        gesture = best?.categoryName ?? 'None';
        score = best?.score ?? 0;
    }
</script>

<h1>First demo</h1>
<CameraStage onload={load} onframe={frame} onstop={() => (gesture = 'None')} />
<p>The model sees: <strong role="status">{gesture}</strong> ({Math.round(score * 100)} %)</p>

4. Run it. npm run dev, open localhost:5173/demo in Chrome or Edge, press Turn camera on, allow the camera, and show a thumbs-up, a fist, a V. The line under the video changes.

Find the four boxes of the pipeline in those lines:

Then press Turn camera off and look at the light next to your webcam. It goes out, because CameraStage calls track.stop() on every track of the stream. Hiding the video would not do that.

If something goes wrong¶

Want more? Set numHands: 2 in vision.ts and show result.gestures with an {#each}.

The states of a camera¶

A camera can fail in ways a button never does, and people worry about it more than about anything else on the page. So the component has states, and each one tells the user what is going on. Break your demo on purpose and see them:

camera.ts also has messages for a camera that another app is using and for a page that is not in a secure context.

The "Blocked" message is Nielsen's heuristic number 9: help users recognise, diagnose and recover from errors. The line "never uploaded" is visibility of system status, about the thing people fear most.

Key takeaways¶

• A natural user interface is operated by speaking and moving. It comes after the CLI and the GUI, and it is not natural: gestures are invisible, so it needs a visible vocabulary and feedback.
• There are four families: voice, hands and face, whole body and augmented reality.
• Every NUI is a pipeline: sensor → model → rule → feedback. The model gives a label and a score. The rule is your code.
• The toolbox: MediaPipe Tasks, Web Speech API, WebXR, TensorFlow.js, ONNX Runtime Web and Transformers.js.
• MediaPipe runs in the browser: frames are not uploaded by the library, at the price of a first download and of your computer's power. "Never uploaded" is a promise your own code has to keep.
• A camera needs a secure context and clear states, and turning it off means calling track.stop().

Quick check¶

Your app can see. Next it gets an ear and a mouth, and it learns to say "I did not understand": Voice commands.