Hands
A gesture-controlled digital theremin that transforms hand movements into musical expression through computer vision and real-time audio synthesis.
Hands
Project Summary
hands is an innovative gesture-controlled digital theremin that transforms hand movements into musical expression through computer vision and real-time audio synthesis. It's a web-based interactive music instrument that combines cutting-edge AI hand tracking with sophisticated audio processing.
Creative Purpose
The project reimagines the traditional theremin (an electronic musical instrument played without physical contact) for the digital age. It transforms the user's body into a musical interface, allowing them to:
- Create music through natural hand gestures - no physical instruments required
- Express musical ideas through movement - turning dance and gesture into sound
- Explore new forms of musical interaction - combining visual art, music, and technology
- Make music accessible - anyone can create sounds by simply moving their hands
Technologies Used
Frontend Framework
- Next.js 15.4.4 - Modern React framework with App Router
- React 19.1.0 - Latest React with concurrent features
- TypeScript 5 - Type-safe development
Computer Vision & AI
- MediaPipe Tasks Vision - Google's AI framework for hand landmark detection
- TensorFlow.js - Machine learning capabilities in the browser
- Real-time hand tracking with 21-point hand landmark detection
- Gesture recognition for pinch, high-five, and finger combinations
3D Graphics & Visualization
- Three.js 0.178.0 - 3D graphics library
- React Three Fiber - React renderer for Three.js
- React Three Drei - Useful helpers and controls
- Custom shaders for real-time visual effects
Audio Synthesis & Processing
- Tone.js 15.1.22 - Advanced Web Audio API framework
- Multiple synthesis modes (warm, bright, soft, aggressive)
- Real-time audio effects (filter, distortion, chorus, reverb)
- Musical scale quantization (major, minor, pentatonic, blues, dorian)
- Dynamic octave control and frequency modulation
Styling & UI
- Tailwind CSS 4 - Utility-first CSS framework
- Responsive design with modern UI components
- Real-time feedback and visual indicators
Development & Deployment
- ESLint - Code quality and consistency
- Fly.io - Containerized deployment platform
- Docker - Containerization support
Key Features
Dual-Hand Control System
- Right Hand: Controls note selection (X-axis), filter cutoff (Z-axis), and octave changes (pinch gesture)
- Left Hand: Controls volume (palm rotation) and sound modifiers (finger combinations)
Advanced Gesture Recognition
- Pinch gestures for different sound effects
- High-five detection for resetting to normal mode
- Hand rotation tracking for volume control
- Real-time gesture feedback
Musical Intelligence
- Scale quantization to ensure musical notes
- Multiple musical scales and root note selection
- Dynamic octave shifting
- Professional audio effects chain
Immersive Visualization
- 3D reactive graphics that respond to hand movement
- Real-time frequency visualization
- Gesture-aware visual effects
- Interactive 3D scene with orbit controls
Technical Architecture
The project follows a modern React architecture with:
- Custom hooks for hand tracking and audio synthesis
- Component-based design for modularity
- Real-time data flow from camera to audio to visualization
- Performance optimization with React 19 concurrent features
- Type-safe development throughout the codebase
This project represents a fusion of AI, music technology, and interactive art, creating an entirely new way to experience and create music through natural human movement.