Pose detection: 33 points and a referee¶

Exercise pose-detection-annex.pdf · PDF

Your hands are covered in flour and the recipe on the laptop has just scrolled off the screen. You raise your right hand. The next step appears. You keep your hand up and count to three. How many steps should it advance? One, of course. Getting that "one" right is the whole trick of this page.

In 2010, doing this took a Kinect, a special depth camera that sold 8 million units in 60 days. Now you will do it with a webcam. In the last camera demo the model told you a word: Thumb_Up. Here it tells you nothing but 33 points. The meaning is all yours.

The map of the body¶

A landmark is a numbered point the model finds on the body. A pose landmarker is the model that finds them: MediaPipe Pose Landmarker returns 33 landmarks for each person, on every frame.

flowchart TB
    N["<b>0</b> nose"] --> LS["<b>11</b> left shoulder"]
    N --> RS["<b>12</b> right shoulder"]
    LS --> LE["<b>13</b> left elbow"] --> LW["<b>15</b> left wrist"]
    RS --> RE["<b>14</b> right elbow"] --> RW["<b>16</b> right wrist"]
    LS --> LH["<b>23</b> left hip"] --> LK["<b>25</b> left knee"] --> LA["<b>27</b> left ankle"]
    RS --> RH["<b>24</b> right hip"] --> RK["<b>26</b> right knee"] --> RA["<b>28</b> right ankle"]

The full list:

From 7 on, odd is left and even is right. And "left" is the person's own left.

Reading the coordinates¶

Draw a rectangle and mark the point (0, 0). If you pointed at the bottom-left corner, as in maths class, you have just found the most common bug on this page. On a screen, the origin is at the top-left corner.

So if your nose is at y = 0.25 and your right wrist at y = 0.1, your hand is above your nose: 0.1 is nearer the top.

The numbers are fractions of the picture, not pixels. A value of 0.5 is the middle, whatever the size of the camera image. MediaPipe also returns worldLandmarks, in metres, with the origin at the hips. You will not need them for this page.

A pose is a rule¶

Write the pose in words, then in code. "The right hand is raised" means "the right wrist is above the right shoulder":

Look at the squat: the hips are lower than the knees, so their y is bigger.

And use names, not magic numbers. RIGHT_WRIST tells the next reader what 16 never will.

The referee¶

Think of a football referee. The referee watches the ball all the time: on every frame. While the ball is out, it is out for several seconds. That is a state. But the whistle sounds once, at the moment the ball crosses the line. That is an event.

A state lasts: the hand is up. An event happens once: the hand went up.

A referee who whistled for as long as the ball was out would be unbearable. So would your app: the rule is true on 30 frames per second, so if (handUp) box.next() jumps 30 notes per second.

The referee needs a memory: what was true on the previous frame?

const RIGHT_SHOULDER = 12, RIGHT_WRIST = 16; // names, not magic numbers
let rightUp = false; // memory: was the hand up in the previous frame?

// ...inside frame():
const rightNow = pose[RIGHT_WRIST].y < pose[RIGHT_SHOULDER].y; // above = smaller y
if (rightNow && !rightUp) box.next(); // the whistle: only when it BECOMES true
rightUp = rightNow; // remember for the next frame

An edge trigger fires on the change from false to true, the "edge" between the two, and not while the condition stays true.

Why is rightUp a plain let and not $state? Nothing on the screen reads it. It is the referee's memory, not something to paint. It would still work with $state, but $state is for what the screen shows.

Your pose page¶

You need the camera demo of the first page working in your project, and the box and NoteList from Voice commands, or functions of your own app. Before you write code, write your two poses as a table: rule, action, feedback. The actions must be functions your app already has.

1. Add the task to vision.ts. It is the same pattern with another task and another model (5.8 MB). Add PoseLandmarker to the import at the top.

import { FilesetResolver, GestureRecognizer, PoseLandmarker } from '@mediapipe/tasks-vision';

export async function createPoseLandmarker() {
    return PoseLandmarker.createFromOptions(await vision(), {
        baseOptions: {
            modelAssetPath: `${MODELS}/pose_landmarker/pose_landmarker_lite/float16/latest/pose_landmarker_lite.task`,
            delegate: 'GPU'
        },
        runningMode: 'VIDEO',
        numPoses: 1
    });
}

2. Create the page, src/routes/pose/+page.svelte. Each rule gets its own edge trigger, with its own memory.

<script lang="ts">
    import { DrawingUtils, PoseLandmarker } from '@mediapipe/tasks-vision';
    import CameraStage from '#lib/components/CameraStage.svelte';
    import NoteList from '#lib/components/NoteList.svelte';
    import { box } from '#lib/notes.svelte.js';
    import { createPoseLandmarker } from '#lib/nui/vision.js';

    // Indexes of the 33 pose landmarks we need (the person's own left and right).
    const LEFT_SHOULDER = 11;
    const RIGHT_SHOULDER = 12;
    const LEFT_WRIST = 15;
    const RIGHT_WRIST = 16;

    let landmarker: PoseLandmarker;
    let seen = $state(false);
    let rightUp = false; // plain variables: they remember the previous frame
    let leftUp = false;

    async function load() {
        landmarker = await createPoseLandmarker();
    }

    function frame(video: HTMLVideoElement, now: number, ctx: CanvasRenderingContext2D) {
        const result = landmarker.detectForVideo(video, now);
        const pose = result.landmarks[0]; // first person: 33 points, x and y from 0 to 1
        seen = pose !== undefined;
        if (!pose) return; // nobody in the picture

        // The skeleton, on the canvas that CameraStage gives you.
        const draw = new DrawingUtils(ctx);
        draw.drawConnectors(pose, PoseLandmarker.POSE_CONNECTIONS, { color: '#ffffff', lineWidth: 3 });
        draw.drawLandmarks(pose, { color: '#d62828', radius: 4 });

        // y grows downwards: "above" means a SMALLER y.
        const rightNow = pose[RIGHT_WRIST].y < pose[RIGHT_SHOULDER].y;
        const leftNow = pose[LEFT_WRIST].y < pose[LEFT_SHOULDER].y;

        // Fire once, on the frame the hand goes up (not 30 times per second while it stays up).
        if (rightNow && !rightUp) box.next();
        if (leftNow && !leftUp) box.previous();
        rightUp = rightNow;
        leftUp = leftNow;
    }
</script>

<h1>Pose</h1>
<CameraStage onload={load} onframe={frame} onstop={() => (seen = false)} />
<p role="status">
    {seen ? 'Raise your right hand for the next note, your left hand for the previous one.' : 'Step back until your shoulders are in the picture.'}
</p>
<NoteList />

Open localhost:5173/pose, turn the camera on and step back. A white skeleton appears over you. Stand about two metres away, with the light in front of you: a window behind you turns you into a silhouette, and the landmarks jump.

If something goes wrong¶

Want more? Build a squat counter that counts once per squat. Or ignore points the model is not sure about, with pose[16].visibility > 0.5. And if the classroom PCs are fast, the full model (9.4 MB, replace pose_landmarker_lite with pose_landmarker_full twice in the URL) is steadier.

The buttons stay¶

A classmate has a broken arm in a cast. Can they still go to the next note in your app? They must: the Next and Previous buttons stay on the page. A pose is a second way to do something, never the only one. Standing two metres from a camera with both arms free is a lot to ask of anyone. The next page gives this rule its official name.

Key takeaways¶

• A pose landmarker returns 33 landmarks per person: shoulders 11, 12, wrists 15, 16, hips 23, 24. Odd is left and even is right from 7 on, and left is the person's left.
• Coordinates go from 0 to 1 with the origin at the top-left, so up means a smaller y. visibility says how likely it is that the point is in view.
• There is no list of poses. A pose is a rule: a comparison you write between landmarks.
• A state lasts and an event happens once. An edge trigger fires only on the change from false to true, using a memory of the previous frame.
• Memory between frames is a plain variable. $state is for what the screen shows.
• A pose is a second way in. The buttons stay.

Quick check¶

You can now read a whole body. Next you zoom in on hands and faces, and you stop a gesture from firing when nobody meant it: Gestures, faces and AR.