GestureCanvas

Overview

An infinite canvas driven by bare hands instead of a mouse. A webcam feed runs through MediaPipe Hand Landmarker in the browser, and a gesture engine turns 21 landmarks per hand into a vocabulary of 9 gestures: point to move the cursor, pinch to draw or drag, two fingers to connect elements, a held fist to delete, a pinch-hold to open a radial menu. The hard part is not detection but reliability on a noisy webcam feed, so the engine layers One-Euro filtering, hysteresis bands, asymmetric debouncing, and dwell-to-commit on every action. Every threshold is fixed in a written gesture contract that the implementation must match, and an optional command bar turns natural language into canvas operations, degrading to gesture-only when no key is configured.

Highlights

• 9-gesture vocabulary: point, pinch, middle-pinch, pinch-hold, two-hand shape, connect, fist-delete, thumbs-up, open palm
• One-Euro filtering with beta retuned roughly 3 orders of magnitude for MediaPipe's normalized coordinate space
• Hysteresis plus asymmetric debouncing, so in-pinch jitter never breaks a stroke but a real release disengages instantly
• Dwell-to-commit gates every destructive or creative action, so nothing fires by accident
• 40 test suites plus end-to-end coverage, with the engine tested against synthetic landmarks so detection has regression cover without a webcam

Stack & tools

• Next.js 16
• React 19
• TypeScript
• MediaPipe Tasks Vision
• Three.js / React Three Fiber
• Motion
• Tailwind 4
• Zustand
• AI SDK v6
• Upstash Redis
• Vitest
• Playwright