Voice commands: press to talk, say it back¶

Exercise voice-commands-annex.pdf · PDF

You press a button and say "new note homework". The words appear on the screen while you speak, the note is created, and the computer answers out loud: "Created note Homework." Then you say "open the pod bay doors". The app answers: I heard "open the pod bay doors". That is not a command. Did the ear fail, or the vocabulary?

The ear worked perfectly. Those are two different problems, and keeping them apart is what this page is about. In the last page your app opened its eyes. Now it gets an ear and a mouth, with nothing to install, and it learns the most useful sentence in voice design: "I did not understand."

An ear and a mouth¶

The Web Speech API is the part of the browser that handles voice. It has two separate halves: speech recognition turns speech into text, and speech synthesis turns text into speech.

Which browsers?¶

The two halves are not equally supported. This table decides which browser you test in (checked against MDN's compatibility data, November 2026):

So: use Chrome or Edge 139+ for this unit. The mouth works almost everywhere. And browser support changes while you are reading this, so your code must never assume: it asks "do you have an ear?" before using it.

Where does your voice go?¶

The video in the last page was analysed in the tab. When you press Speak in Chrome, where does your voice go?

By default, to a server. MDN puts it bluntly: "Your audio is sent to a web service for recognition processing, so it won't work offline." Two consequences: without internet you get the error network, and your words leave the room.

Since Chrome 139 a page can ask for on-device recognition instead, with recognition.processLocally = true. Then neither the audio nor the transcript goes to a third-party service, at the price of a one-time language pack download per language.

// Returns a recognition that never leaves the device, or null if that is not possible here.
export async function localRecognition(lang: string) {
    if (typeof SpeechRecognition === 'undefined' || !('available' in SpeechRecognition)) return null;
    const options = { langs: [lang], processLocally: true };
    const availability = await SpeechRecognition.available(options);
    if (availability === 'unavailable') return null;
    if (availability !== 'available' && !(await SpeechRecognition.install(options))) return null;

    const recognition = new SpeechRecognition();
    recognition.lang = lang;
    recognition.processLocally = true;
    return recognition;
}

Radio rules¶

Think of a walkie-talkie. Radio operators follow three rules, because they can never be sure the other side heard correctly. Your component follows the same three:

The first rule is also a privacy rule. An app that listens all the time is an app nobody trusts. Setting continuous = true to "get rid of the button" would keep the microphone open, and the app would end up hearing its own answers.

A recognition is a short story¶

A recognition is told by events:

flowchart LR
    B["<b>Press Speak</b><br>start() is called"] --> S["<b>start</b><br>microphone open"]
    S --> R["<b>result</b><br>several times, words so far<br>(the last has isFinal)"]
    S --> E["<b>error</b><br>event.error has a code"]
    R --> N["<b>end</b><br>always the last one"]
    E --> N

The text is at event.results[i][0].transcript: a list of results, each with a list of alternatives. [0] is the most likely one.

Write the vocabulary first¶

A voice vocabulary is the table of everything your app understands: what the user says (several phrasings), what the app does, and what it answers.

Delete answers "Say undo to bring it back" for a reason: this is forgiveness, from Apple's guidelines. A misheard word must never cost a note.

The rules for writing yours:

• 2 to 4 words per command. Short words are misheard: "next note", not "n".
• No two commands that sound alike, such as "delete" and "repeat".
• One destructive command at most, with an Undo row.
• Every command already exists as a button. Voice is a second door, not the only one.

Then say your commands aloud, fast. If you stumble, the ear will too.

The ear¶

Plain JavaScript projects can skip this step. TypeScript's built-in types do not include SpeechRecognition yet, so add them with one install and one line:

npm install -D @types/dom-speech-recognition

// src/app.d.ts, first line
/// <reference types="dom-speech-recognition" />

Now ask the browser for its ear, in src/lib/nui/speech.ts:

// Chrome, Edge 139+: SpeechRecognition. Older Chromium, Safari: webkitSpeechRecognition.
// A browser with neither gets undefined, and the page offers the typed fallback.
export function recognitionClass(): typeof SpeechRecognition | undefined {
    if (typeof window === 'undefined') return undefined; // server-side rendering
    const w = window as typeof window & { webkitSpeechRecognition?: typeof SpeechRecognition };
    return w.SpeechRecognition ?? w.webkitSpeechRecognition;
}

Why the typeof window check? SvelteKit also renders the page on the server, and a server has no window and no microphone.

Your first transcript needs only this, in a new VoiceButton.svelte:

<script lang="ts">
    import { recognitionClass } from '#lib/nui/speech.js';
    let heard = $state('');

    function listen() {
        const Recognition = recognitionClass();
        if (!Recognition) return; // later, this becomes a real state
        const recognition = new Recognition();
        recognition.lang = 'en-US'; // or 'es-ES', 'ca-ES': never let the browser guess
        recognition.continuous = false; // one sentence, then stop
        recognition.interimResults = true; // show the words while they are spoken
        recognition.onresult = (event) => {
            heard = event.results[event.results.length - 1][0].transcript;
        };
        recognition.onerror = (event) => console.log(event.error);
        recognition.start(); // the browser asks for the microphone
    }
</script>

<button type="button" onclick={listen}>Speak</button>
<p>{heard}</p>

See your own words on the screen before you go on. No microphone? Borrow one for this step only: from the next section on, you can work with the keyboard.

From words to a command¶

The ear returns free text: "New note, shopping list." Your app needs a decision. Put that decision in a pure function: a function whose answer depends only on what you give it, with no microphone inside and no side effects. Then you can test it with Vitest and feed it from a text box.

src/lib/nui/commands.ts:

export type Command =
    | { type: 'new'; title: string }
    | { type: 'delete' }
    | { type: 'undo' }
    | { type: 'next' }
    | { type: 'previous' };

export function parseCommand(transcript: string): Command | null {
    // "New note, shopping list." -> "new note shopping list"
    const text = transcript.toLowerCase().replace(/[.,!?]/g, '').replace(/\s+/g, ' ').trim();

    const created = text.match(/^(?:new|add|create) note(?: called)?(?: (.+))?$/);
    if (created) return { type: 'new', title: created[1] ?? '' };
    if (/^(delete|remove)( this| the)? note$/.test(text)) return { type: 'delete' };
    if (/^undo$/.test(text)) return { type: 'undo' };
    if (/^next( note)?$/.test(text)) return { type: 'next' };
    if (/^(previous|back)( note)?$/.test(text)) return { type: 'previous' };
    return null; // heard something, but it is not a command
}

Three things to notice:

• Normalise first. The recogniser adds capitals and punctuation, and it makes mistakes ("knew note").
• ^ and $ make the whole sentence match, so "do not delete note" does nothing.
• null is an honest answer. It is the "say again" of the radio.

Because it is pure, the tests are three lines of real value:

import { expect, test } from 'vitest';
import { parseCommand } from './commands';

test('capitals and punctuation do not matter', () => {
    expect(parseCommand('New note, shopping list.')).toEqual({ type: 'new', title: 'shopping list' });
});

test('a sentence that merely contains a command does nothing', () => {
    expect(parseCommand('do not delete note')).toBeNull();
});

The typed fallback¶

One handle(transcript) function, two ways in: the final result of the ear, and a small form.

<form onsubmit={submitTyped}>
    <label for="typed-command">Type a command</label>
    <input id="typed-command" bind:value={typed} autocomplete="off" />
    <button type="submit">Run</button>
</form>

Six states¶

Like the camera, the voice button has states, and the user must always know which one it is in.

And one message per error code, in speech.ts:

Two pieces of code matter most. First, support is checked in $effect, which runs only in the browser, after the component is mounted. Checked at the top of the script, it would run on the server too, where there is no window. The same $effect releases the microphone when you leave the page. Second, the message sits inside <div role="status" aria-live="polite">, so a screen reader announces every change (WCAG 4.1.3).

Common mistakes:

• The state is shown only by colour. The red button must also change its word.
• The button stays on "Listening" after silence. Use end: if the status is still listening, go back to idle.
• Clicking twice throws InvalidStateError. start() cannot be called twice. Disable the button while listening.

The mouth¶

Add this to speech.ts:

export function speak(text: string, lang = 'en-US') {
    if (typeof window === 'undefined' || !('speechSynthesis' in window)) return;
    window.speechSynthesis.cancel(); // do not queue behind an old sentence
    const utterance = new SpeechSynthesisUtterance(text);
    utterance.lang = lang;
    utterance.rate = 1; // 0.1 to 10; pitch 0 to 2; volume 0 to 1
    window.speechSynthesis.speak(utterance);
}

The same sentence is shown and spoken: the user who cannot hear reads it, the user who is not looking hears it. That is the read-back of the radio.

Why do we not speak the "not a command" message? A machine that talks every time it fails is a machine you mute. And why is it safe to speak right after listening? With continuous = false, recognition stops after one final result, so the app does not take orders from its own voice.

The whole component¶

Here is everything together. To try it, you need a small note box, a list with its buttons, and a page.

// What the user reads for each SpeechRecognitionErrorEvent.error code.
export function speechProblem(code: string): string {
    switch (code) {
        case 'not-allowed':
        case 'service-not-allowed':
            return 'Microphone blocked. Allow it from the icon in the address bar, or type the command below.';
        case 'audio-capture':
            return 'No microphone found. Plug one in, or type the command below.';
        case 'network':
            return 'Speech recognition needs the internet in this browser. Type the command below.';
        case 'no-speech':
            return 'I did not hear anything. Press the button and try again.';
        case 'language-not-supported':
            return 'This language is not supported for recognition. Type the command below.';
        default:
            return 'Something went wrong with speech recognition. Type the command below.';
    }
}

<script lang="ts">
    import { parseCommand, type Command } from '#lib/nui/commands.js';
    import { recognitionClass, speak, speechProblem } from '#lib/nui/speech.js';

    type Status = 'idle' | 'listening' | 'recognised' | 'unknown' | 'error' | 'unsupported';

    type Props = {
        /** Runs the command and returns the sentence to show and say back. */
        oncommand: (command: Command) => string;
        lang?: string;
    };
    let { oncommand, lang = 'en-US' }: Props = $props();

    const hint = 'Press Speak and say a command, for example "new note homework".';

    let status = $state<Status>('idle');
    let heard = $state('');
    let message = $state(hint);
    let typed = $state('');
    let recognition: SpeechRecognition | undefined;

    // Runs in the browser only (never during server-side rendering), once the component is mounted.
    $effect(() => {
        if (!recognitionClass()) {
            status = 'unsupported';
            message = 'This browser has no speech recognition. Type the command below.';
        }
        return () => recognition?.abort(); // leaving the page: release the microphone
    });

    function handle(transcript: string) {
        heard = transcript;
        const command = parseCommand(transcript);
        if (!command) {
            status = 'unknown';
            message = `I heard "${transcript}". That is not a command. Try "new note homework".`;
            return;
        }
        status = 'recognised';
        message = oncommand(command);
        speak(message, lang);
    }

    function listen() {
        const Recognition = recognitionClass();
        if (!Recognition || status === 'listening') return;

        recognition = new Recognition();
        recognition.lang = lang;
        recognition.continuous = false; // one sentence, then stop: push-to-talk
        recognition.interimResults = true; // show the words while they are being spoken

        recognition.onstart = () => {
            status = 'listening';
            heard = '';
            message = 'Listening...';
        };
        recognition.onresult = (event) => {
            const result = event.results[event.results.length - 1];
            heard = result[0].transcript;
            if (result.isFinal) handle(heard);
        };
        recognition.onerror = (event) => {
            status = 'error';
            message = speechProblem(event.error);
        };
        recognition.onend = () => {
            // Ended while still "listening": nothing final arrived.
            if (status === 'listening') {
                status = 'idle';
                message = hint;
            }
        };
        recognition.start(); // the browser asks for the microphone the first time
    }

    function submitTyped(event: SubmitEvent) {
        event.preventDefault();
        if (typed.trim()) handle(typed);
        typed = '';
    }
</script>

<section class="voice" data-status={status}>
    <button
        type="button"
        class="mic"
        onclick={listen}
        disabled={status === 'unsupported' || status === 'listening'}
    >
        {status === 'listening' ? 'Listening' : 'Speak'}
    </button>

    <!-- Announced by screen readers every time it changes -->
    <div class="status" role="status" aria-live="polite">
        {#if heard && status === 'listening'}<p class="heard">{heard}</p>{/if}
        <p>{message}</p>
    </div>

    <!-- The same vocabulary without a microphone: fallback and accessibility in one -->
    <form onsubmit={submitTyped}>
        <label for="typed-command">Type a command</label>
        <input id="typed-command" bind:value={typed} autocomplete="off" />
        <button type="submit">Run</button>
    </form>
</section>

<style>
    .voice {
        display: grid;
        gap: 0.75rem;
        border: 3px solid #111;
        padding: 1rem;
    }
    .voice[data-status='listening'] .mic {
        background: #d62828;
        color: #fff;
    }
    .heard {
        font-style: italic;
    }
    .status p {
        margin: 0;
    }
</style>

// NoteBox state: one shared, reactive box of notes (Svelte 5 runes in a .svelte.ts module).
export type Note = { id: number; title: string; body: string; pinned: boolean };

class NoteBox {
    notes = $state<Note[]>([
        { id: 1, title: 'Shopping list', body: 'Milk, eggs, bread.', pinned: false },
        { id: 2, title: 'Recipe', body: 'Two eggs, a pinch of salt.', pinned: true },
        { id: 3, title: 'Ideas', body: 'An app you can talk to.', pinned: false }
    ]);
    index = $state(0);
    lastDeleted = $state<Note | null>(null);

    current = $derived<Note | undefined>(this.notes[this.index]);

    #nextId = 4;

    add(title: string) {
        this.notes.push({ id: this.#nextId++, title: title || 'Untitled', body: '', pinned: false });
        this.index = this.notes.length - 1;
    }

    // Forgiveness: delete at once, keep the note for Undo.
    remove() {
        if (!this.current) return;
        this.lastDeleted = this.current;
        this.notes.splice(this.index, 1);
        this.index = Math.max(0, Math.min(this.index, this.notes.length - 1));
    }

    undo() {
        if (!this.lastDeleted) return;
        this.notes.push(this.lastDeleted);
        this.index = this.notes.length - 1;
        this.lastDeleted = null;
    }

    next() {
        if (this.notes.length) this.index = (this.index + 1) % this.notes.length;
    }

    previous() {
        if (this.notes.length) this.index = (this.index - 1 + this.notes.length) % this.notes.length;
    }

    togglePin() {
        if (this.current) this.current.pinned = !this.current.pinned;
    }
}

export const box = new NoteBox();

<script lang="ts">
    import { box } from '#lib/notes.svelte.js';
</script>

<section aria-label="Notes">
    <ul>
        {#each box.notes as note, i (note.id)}
            <li>
                <button type="button" aria-current={i === box.index} onclick={() => (box.index = i)}>
                    {note.title}{note.pinned ? ' (pinned)' : ''}
                </button>
            </li>
        {:else}
            <li>No notes yet.</li>
        {/each}
    </ul>

    {#if box.current}
        <article>
            <h2>{box.current.title}</h2>
            <textarea bind:value={box.current.body} aria-label="Note text" rows="4"></textarea>
        </article>
    {/if}

    <!-- Every voice or gesture action also exists as a button -->
    <button type="button" onclick={() => box.add('')}>New</button>
    <button type="button" onclick={() => box.previous()}>Previous</button>
    <button type="button" onclick={() => box.next()}>Next</button>
    <button type="button" onclick={() => box.togglePin()} disabled={!box.current}>Pin</button>
    <button type="button" onclick={() => box.remove()} disabled={!box.current}>Delete</button>

    {#if box.lastDeleted}
        <p role="status">
            {box.lastDeleted.title} deleted
            <button type="button" onclick={() => box.undo()}>Undo</button>
        </p>
    {/if}
</section>

<script lang="ts">
    import NoteList from '#lib/components/NoteList.svelte';
    import VoiceButton from '#lib/components/VoiceButton.svelte';
    import { box } from '#lib/notes.svelte.js';
    import type { Command } from '#lib/nui/commands.js';

    // Runs the command and returns the sentence the app says back.
    function run(command: Command): string {
        switch (command.type) {
            case 'new': {
                const title = command.title.charAt(0).toUpperCase() + command.title.slice(1);
                box.add(title);
                return `Created note ${box.current?.title}.`;
            }
            case 'delete': {
                const title = box.current?.title;
                if (!title) return 'There is no note to delete.';
                box.remove();
                return `Deleted ${title}. Say undo to bring it back.`;
            }
            case 'undo':
                if (!box.lastDeleted) return 'Nothing to undo.';
                box.undo();
                return `Restored ${box.current?.title}.`;
            case 'next':
                box.next();
                return box.current?.title ?? 'No notes.';
            case 'previous':
                box.previous();
                return box.current?.title ?? 'No notes.';
        }
    }
</script>

<h1>Voice</h1>
<p>In Chrome, what you say is sent to a recognition service to turn it into text. The microphone only listens after you press Speak.</p>
<VoiceButton oncommand={run} />
<NoteList />

Swap seats with a classmate: can they run two of your commands using only what the screen says? Then block the microphone from the address bar and press Speak. If the message tells them what to do, you are done.

Key takeaways¶

• The Web Speech API has two halves: speech recognition (SpeechRecognition, the ear) and speech synthesis (speechSynthesis, the mouth). Listening needs permission; speaking does not.
• Chrome and Edge 139+ have it unprefixed, Safari only as webkitSpeechRecognition, and Firefox is only now arriving on desktop. Always check before you use it.
• In Chrome, recognition goes to a server by default: no internet, no ear. processLocally = true keeps it on the device, after a language pack download. Say on the page which one you use.
• Radio rules: press to talk, read back, "say again". Never act on a guess.
• A voice vocabulary lists what the user says, what the app does and what it answers. Every command also exists as a button.
• A pure function turns the transcript into a command, or null. It feeds both the ear and the typed fallback, which stays for good.

Quick check¶

The ear and the mouth are working. Next the app watches your whole body, so wear something you can raise your arms in: Pose detection.