Skip to content

Unit 5 · Natural user interfaces

Not every interface needs a mouse. Here the app reacts to your voice, your hands and your body.

Learning outcome Hours Term
RA2 — Generates natural user interfaces using visual tools. 10 2nd

What you will learn

  • Identify the machine learning tools available for user interfaces
  • Create a natural user interface with those tools
  • Use speech recognition to trigger actions
  • Use body movement detection to trigger actions
  • Use detection of body parts (hands, face...) to trigger actions
  • Add augmented reality to a user interface

Lessons

  1. Natural interfaces — what a NUI is, its four families, the pipeline sensor → model → rule → feedback, the ML toolbox, and a first webcam gesture demo
  2. Voice commands — speech recognition and synthesis with the Web Speech API: browser support, where the audio goes, a voice vocabulary, a parser and a typed fallback
  3. Pose detection — the 33 body landmarks, coordinates where y grows downwards, poses as rules, and the edge trigger that fires an action once
  4. Gestures, faces, AR — hold-to-confirm gestures, face detection versus recognition, an AR sticker on a mirrored video, WebXR support, and a privacy and accessibility audit