Visual tests and documentation¶

Exercise visual-tests-docs-annex.pdf · PDF

Remember those puzzles with two nearly identical drawings? You get 60 seconds to spot the differences. Most people find three or four, and there are usually more. Your eyes are good, but they get tired, and they do not check every pixel.

Now delete the whole <style> block from Button.svelte. Storybook shows a grey browser-default button. Run npm test: ten passed. How? We have a test called "is primary and medium by default". It checks that the class is set. The class is still there. The CSS behind it is gone. This page gives the machine eyes, and then writes the instructions for the box.

Eyes for the machine¶

A visual regression test takes a screenshot of a component in a real browser and compares it, pixel by pixel, with an approved screenshot. A regression is something that used to be right and no longer is.

The machine from the unit tests page checks that the brick grips. This one checks that it is the right colour and shape, against a photo of a perfect one.

Three images, three names:

flowchart LR
    B["<b>Baseline</b><br>the approved photo<br>(committed to git)"] --> C{"compare<br>pixel by pixel"}
    A["<b>Actual</b><br>the photo taken now"] --> C
    C -->|"no difference"| G["Test passes"]
    C -->|"different"| D["<b>Diff</b><br>the changed pixels,<br>in red"]

Playwright on the stories¶

Playwright drives a real browser (Chromium, Firefox, WebKit) from code: go to a URL, press a key, take a screenshot. Microsoft released it on 31 January 2020.

And we already have a URL for every component in every state: each story has its own address. The catalogue becomes the test bench.

npm install -D @playwright/test
npx playwright install chromium

The second command downloads a browser, so it takes a while, and you need it once per computer. Then a new file next to package.json:

import { defineConfig } from '@playwright/test';

export default defineConfig({
    testDir: 'e2e',
    // 'list' prints to the terminal; 'html' writes playwright-report/ for `npx playwright show-report`.
    reporter: [['list'], ['html', { open: 'never' }]],
    use: { baseURL: 'http://localhost:6006' },
    // How different two pixels may be and still count as equal: 0 = identical, 1 = anything goes.
    // Playwright's default is 0.2, which lets a red button turn orange. We are stricter.
    expect: { toHaveScreenshot: { threshold: 0.05 } },
    webServer: {
        command: 'npm run storybook -- --ci',
        url: 'http://localhost:6006',
        reuseExistingServer: true,
        timeout: 120_000
    }
});

webServer means Playwright starts Storybook by itself if it is not already running, and waits for port 6006. Add test-results and playwright-report to .gitignore, and a script to package.json: "test:visual": "playwright test".

The test goes in a new folder, e2e/. Open a story and click Open canvas in new tab: that URL is what we visit.

import { expect, test } from '@playwright/test';

// One story = one URL. The id comes from the story's title and name:
// "NoteBox/Button" + "Primary" -> notebox-button--primary
const stories = ['primary', 'secondary', 'danger', 'small', 'disabled'];

for (const story of stories) {
    test(`Button · ${story} looks like its baseline`, async ({ page }) => {
        await page.goto(`/iframe.html?id=notebox-button--${story}&viewMode=story`);

        await expect(page.locator('#storybook-root')).toHaveScreenshot(`button-${story}.png`);
    });
}

test('Button shows a focus ring when reached with Tab', async ({ page }) => {
    await page.goto('/iframe.html?id=notebox-button--primary&viewMode=story');
    const button = page.getByRole('button', { name: 'Save' });
    await expect(button).toBeVisible(); // the keyboard does not wait for the story: we do

    await page.keyboard.press('Tab');

    await expect(button).toBeFocused();
    await expect(page).toHaveScreenshot('button-primary-focus.png');
});

• The same three steps as before. Arrange: goto. Act: nothing, or a key press. Assert: toHaveScreenshot.
• getByRole('button', { name: 'Save' }) is the same idea as in Testing Library, on purpose.
• The last test is one jsdom could never do: a real Tab, a real focus ring, photographed. It takes a photo of the whole page, because the ring is drawn outside the button and the edges of #storybook-root would cut it off.

Run it twice¶

Run npx playwright test. First run: six failed.

Error: A snapshot doesn't exist at .../e2e/button.visual.spec.js-snapshots/button-primary-linux.png, writing actual.

That is not a bug. There was nothing to compare with, so Playwright wrote the baseline. Open the new folder e2e/button.visual.spec.js-snapshots/: six PNG files. Look at them. You are approving them. Run it again: six passed, in a few seconds.

Red: bug or decision?¶

Change one line in tokens.css: --nb-radius: 0px. Run the tests: six failed, and each one says something like this:

Error: expect(locator).toHaveScreenshot(expected) failed
  53 pixels (ratio 0.01 of all image pixels) are different.
Expected: e2e/button.visual.spec.js-snapshots/button-primary-linux.png
Received: test-results/.../button-primary-actual.png
Diff:     test-results/.../button-primary-diff.png

Open npx playwright show-report: expected, actual and diff side by side, with red pixels at the four corners. One token, six failures. Good or bad? Good: it shows how far one line reaches.

A red visual test is a question, not a verdict:

Two honest limits:

• Platform. The files end in -linux.png. A screenshot taken on Windows or macOS renders fonts differently and will not match. The Playwright docs say to run the tests in the same environment where the baselines were generated.
• Tolerance. threshold sets how different one pixel may be, from 0 (strict) to 1 (anything goes). At the default of 0.2, a Delete button changed from #dc2626 red to #ea580c orange still passes. That is why our config says 0.05. Even then, a very small colour change can slip through. The machine's eyes are good, not perfect.

Which test for which promise?¶

Behaviour goes to a unit test. Appearance goes to a visual test. You need both: a screenshot cannot click Undo or wait five seconds.

The page nobody writes¶

A new teammate joins tomorrow and needs a button with a red background. How do they find out that <Button variant="danger"> exists? They read the source. Or they ask you. Or, most likely, they write a new one. An undocumented component gets rebuilt.

Every LEGO set has an instruction booklet. Nobody at LEGO draws it by hand from memory: it comes from the same model the set was designed in. Our booklet comes from the code. Documentation written apart from the code goes out of date the first time someone is in a hurry. Documentation generated from the code cannot disagree with it.

One line: autodocs¶

Autodocs is the Storybook feature that generates a documentation page for a component from its stories and its JSDoc comments.

Add tags: ['autodocs'] inside defineMeta. A new entry, Docs, appears above the stories.

<script module>
    import { defineMeta } from '@storybook/addon-svelte-csf';
    import { fn } from 'storybook/test';
    import Button from './Button.svelte';

    const { Story } = defineMeta({
        title: 'NoteBox/Button',
        component: Button,
        tags: ['autodocs'],
        argTypes: {
            variant: { control: 'select', options: ['primary', 'secondary', 'danger'] },
            size: { control: 'inline-radio', options: ['md', 'sm'] }
        },
        args: { onclick: fn() }
    });
</script>

<!-- The main action of a screen. One primary per screen; use danger only for destructive actions. -->
<Story name="Primary">Save</Story>

<Story name="Secondary" args={{ variant: 'secondary' }}>New note</Story>

<Story name="Danger" args={{ variant: 'danger' }}>Delete</Story>

<Story name="Small" args={{ variant: 'secondary', size: 'sm' }}>Undo</Story>

<Story name="Disabled" args={{ disabled: true }}>Share</Story>

You already wrote almost all of it:

If the props table is empty or has no descriptions, your JSDoc @typedef is not attached to $props() with /** @type {Props} */.

Write what the type cannot say¶

The type is already in the next column. A description must say what the type cannot: when, why, and what happens at the edges.

To find the right words, ask yourself: when would I not pass this?

A free page: MDX¶

Some things belong to no component: how to import, the rules of the house. For those, there is MDX.

MDX is Markdown that can also use components. Storybook shows an .mdx file as a page in the sidebar.

import { Meta } from '@storybook/addon-docs/blocks';

<Meta title="NoteBox/Introduction" />

# NoteBox UI

The reusable components of the NoteBox app. Every component here has:

- a **Figma component** with the same name and the same properties,
- **stories** for each variant and state,
- **unit tests** next to it (`*.test.js`).

## Use a component

```svelte
<script>
    import { Button } from '#lib';
</script>

<Button variant="danger" onclick={remove}>Delete</Button>
```

## Rules of the house

1. One **primary** button per screen.
2. Unavailable actions are **dimmed, never removed** (`disabled`).
3. Colours and radius come from `tokens.css`. No hex codes inside a component.

If the page does not appear, the file is outside src/, or <Meta title="..." /> is missing.

A site you can send¶

npm run build-storybook

This writes the whole catalogue as static files to storybook-static/. Any web server can host it, and a designer can open a link with nothing installed. The folder is a build output, so it stays out of git. The installer already put it in .gitignore.

Key takeaways¶

• jsdom draws nothing, so unit tests stay green with zero CSS. Visual regression tests look at real pixels.
• A visual test has three images: baseline, actual and diff. The first run only writes the baseline.
• Look at a baseline before committing it. It is a photo somebody approved, not the truth.
• A red visual test is a question. Run --update-snapshots only after reading the diff and deciding the change is wanted.
• Baselines belong to one system, and threshold sets how picky the comparison is.
• Autodocs builds a page from JSDoc and stories. MDX is for pages that belong to no component.

Quick check¶

Your bricks are tested and documented. In Packaging with npm they go in a box that another project can install.