Gesture Glove
Hand movements translated into game controls and MQTT events.
The challenge
Connect physical hand movements to software without a conventional game controller. The prototype needs to distinguish sustained directional input from one-off actions, translate sensor readings into predictable commands, and make those commands available to both local applications and MQTT subscribers.
My contribution
Developed Arduino firmware for a BNO055 motion sensor and a Python bridge that reads its USB serial output. The firmware uses orientation, linear acceleration and angular velocity to produce named gesture events. The bridge parses those events, maps them to keyboard input with pynput and publishes structured messages using paho-mqtt. Two HTML Canvas games provide a visual environment for the resulting controls.
How it works
At startup, the firmware asks the user to hold still and records a reference orientation. It samples the sensor every 25 ms and compares pitch and roll against a 30-degree tilt threshold. Direction changes produce ON/OFF serial messages at 115200 baud. Python converts these into held or released keys and MQTT movement messages. Acceleration and rotation thresholds define separate action events, with a 300 ms cooldown between detections.
Physical movement becomes software input
Sense and calibrate
Arduino reads BNO055 orientation, acceleration and rotation.
Detect gesture events
Thresholds and direction states produce named serial events.
Bridge through Python
USB serial messages are parsed on the computer.
Route the output
Keyboard input drives the demos; MQTT publishes structured events.
Keyboard and MQTT are parallel outputs. The browser demos receive keys, not MQTT messages; a broker is needed for MQTT.
Gesture controls & demos
Tilting forward, backward, left and right maps to W, S, A and D. The Python bridge also maps PUNCH to Space, SHAKE to E, twists to Q/R and flicks to Shift/Ctrl. GloveGame uses WASD for movement, Space for a dash and E/Q/R for gesture feedback. FlightGame uses WASD for steering and speed. Each game implements only the controls it needs.
What it covers
- BNO055 orientation, linear-acceleration and gyroscope readings through Arduino_NineAxesMotion
- Startup reference calibration and threshold-based directional detection
- State tracking sends direction events only when a held direction changes
- USB serial transport between the Arduino and the computer
- Python parsing and keyboard press/release handling with pynput
- MQTT topics: glove/status, glove/gesture, glove/movement, glove/action and glove/raw
- JSON movement and action payloads with timestamps; connection status and disconnect handling
- Two keyboard-driven HTML Canvas demos: a coin-collection game and a flight game
Gallery
Outcome
The source brings together embedded sensing, a desktop input bridge and event-based messaging in one prototype. It includes both browser demos and a separate MQTT subscriber for inspecting published events. The browser games receive keyboard input from the Python bridge; they do not subscribe directly to MQTT.
Prototype scope & limitations
This version uses a USB-connected Arduino and a Python process on the computer; it is not a standalone wireless controller. MQTT requires a running broker, while keyboard control requires the game window to have focus. Gesture recognition uses fixed thresholds rather than a trained model, so sensitivity and action detection need calibration and validation with the physical glove. The gallery contains a schematic overview and source-code excerpts.