a wearable helmet rig that augments hearing and vision.
the work of team HEADLESSMONKEY (kaushal negi, eshwar pingili, and me). two industrial designers and a software dev
what this is
the project started as a question: what does it take to give a person a second pair of ears and a second pair of eyes, on a single raspberry pi 4b, and have the whole thing boot and run reliably in front of a room?
the answer is a helmet. it is 3d-printed, fits on a head, and has a directional microphone array on a pan and tilt mount, a small lcd inside the field of view that enables night vision, runs real-time object detection, a wireless joystick controller, an audio round trip from a usb mic to a pair of bluetooth earbuds, and a cross-team integration that pulls telemetry from another team’s smart shoe and renders it on the same display as the detection feed.
five subsystems. one shared compute platform. zero crashes during the demo. that is the bar we set, and the rest of this blog is how it was built and what it cost us to get there.
the hardware
the full bill of materials:
- compute hub. raspberry pi 4b
- camera. ov5647 (csi)
- lcd. waveshare 1.9 inch st7789v2, 170 by 320, over spi on gpio 25, 27, 18
- servo controller. pca9685 on i2c 0x40, with an external 5v supply
- servos. two sg90-class servos on pca channels 0 and 1, pan and tilt
- microphone. a directional array (jieli technology usb audio) for the listening side
- earbuds. oneplus nord buds 2r over bluetooth
- wireless controller. funduino joystick shield on an esp32 devkit
- ir cut filter. switched by gpio23 for day and night mode
- shell. 3d-printed, two halves, with mounting points for the camera, the lcd, the microphone dish, and the battery
- power. a 4a lipo with a bms, shared ground with the pi, separate 5v rail for the servos
the only user-facing input is the funduino shield. joystick for the head, four buttons for everything else.
architecture, in one diagram
funduino joystick ov5647
| |
| analog + digital | csi
v v
esp32 --------udp 10 hz--- raspberry pi 4b ----spi--- st7789 lcd
| 172.20.10.2:4210 |
| | i2c 0x40
| v
| pca9685 --- servos (pan, tilt)
| |
| | usb audio
| v
| pipewire loopback --- oneplus nord buds 2
|
| bluetooth (separately)
v
bluetooth stack --- sink routing
everything runs as a systemd service. the integration is by signal, by file, and by udp. not by coupling.
the five services
we split the system into five independent daemons. each one has its own lifecycle, its own log stream, and its own restart button. the smart move was to refuse to make any of them aware of the others beyond a single shared interface.
directional_hearing.service
python. the esp32 sends udp packets at 10 hz on port 4210 with the joystick axes and four button states. this daemon receives the packets, parses them, and drives the pca9685 servo channels 0 and 1. hold-position mode: when the stick returns to center, the servos stay where they are. graceful shutdown: on sigterm it releases the servos before exit, so the rig does not snap to a default angle on stop.
the protocol packet looks like this:
X:<pct>,Y:<pct>,DIR:<dir>,A:<0|1>,B:<0|1>,C:<0|1>,D:<0|1>
joystick center is auto-calibrated at boot over 50 samples, with a 250-unit deadzone. we map to a percentage range of negative 100 to 100 instead of raw adc counts, because raw counts drift with temperature and we did not want the calibration to walk.
camera-preview-lcd.service
c++. captures frames from the ov5647, runs yolofastestv2 through ncnn, and renders the result onto the waveshare st7789 lcd over spi. roughly 10 fps at a usable confidence threshold. the frame is rotated 180 degrees in software because the camera is mounted upside down in the shell. a one-slot queue in the capture thread drops stale frames and lifts the steady state from a stuttery 4 fps to a usable 9 to 16 fps depending on the detection load.
two unix signals control the on-screen ui:
sigusr1toggles the fps overlay and the smart shoe telemetry overlay togethersigusr2toggles the bounding boxes
those are wired to buttons on the joystick shield, which is what lets the wearer turn the overlays on and off without taking the helmet off.
smart-shoe-bridge.service
python. the other team’s smart shoe esp32 runs a small web server that exposes activity state, speed, and lace tension at /data. this daemon polls that endpoint at 1 hz and writes the result to /tmp/shoe_overlay.txt in a fixed format. the c++ hud binary reads the file at render time and draws the values at the bottom of the lcd. the win here is that adding a new field to the shoe’s payload does not require recompiling the hud. you change the writer, the reader picks it up.
this is the only piece of the system we did not own, and it is the one we spent the least time on.
bluetooth-audio.service
bash. powers on bluetooth, connects to the oneplus nord buds 2r by mac address, and sets them as the default pipewire sink. if the buds are out of range at boot, it scans continuously until it finds them. the one trap is that pipewire and wireplumber are user session services, so this unit has to set xdg_runtime_dir=/run/user/1000 to find them. without that line, the sink does not exist at boot and the script silently does nothing.
audio-loopback.service
bash. discovers the usb source and the bluetooth sink via wpctl, launches pw-loopback, and auto-restarts if either side disconnects. about 50 ms of latency end to end, which is low enough that the wearer can hear what the directional array is pointing at, in real time, in their ears, without the round trip feeling like a delay.
the control mapping
the funduino shield is the entire user interface. one joystick, four buttons.
| control | function | target |
|---|---|---|
| joystick x | pan servo | pca9685 channel 0 |
| joystick y | tilt servo | pca9685 channel 1 |
| button a | ir cut filter on or off | gpio23 |
| button b | detection bounding boxes on or off | sigusr2 to the detection binary |
| button c | fps overlay and shoe overlay | sigusr1 to the detection binary |
| button d | reserved for future | n/a |
button d is a real decision, not a placeholder. we left it intentionally because the rig was still in motion the week before the demo and we did not want to commit a fourth signal mapping before the system was stable.
the smart shoe integration, in more detail
the seam between our helmet and the other team’s shoe is one file. that is the entire api.
# /tmp/shoe_overlay.txt
S:1.42
W:STANDING
T:62
three values, one line, polled at 1 hz. the c++ binary reads the file inside the render loop, parses the three fields, and draws them at the bottom of the lcd: left is the activity state, middle is the speed in m/s, right is the lace tension as a percentage.
what we got out of treating the integration as a file:
- no recompile to add a new field
- the shoe team can change their payload format and the bridge can adapt without us
- the hud can be tested without the shoe running at all (just write a static file)
- the integration has a clean failure mode: if the file is missing, the overlay is silent
if you take one architectural pattern from this project, take this one. the boundary is what kept the integration to about a day of work and zero regressions in our existing pipeline.
the hard problems, in order of how much sleep they cost us
1. spi lcd frame stalls
the c++ render loop pulled frames faster than the spi bus could push them, so the lcd froze every few seconds. the fix was a one-slot queue in the capture thread. the renderer always reads the most recent frame, never a stale one, and the queue drops anything older. this lifted the steady state from 4 fps to 9 to 16 fps depending on the detection load. the full diagnosis is in the lcd troubleshooting report in the repo.
2. camera orientation
the camera had to be mounted upside down to fit the shell. rotating in software at render time beat rotating the optics. both the python preview and the c++ binary take a --rotate flag, and the systemd unit sets detect_rotate=1 so the boot path is the only source of truth. the hdmi preview script also applies --rotation 180 to rpicam-hello.
3. pipewire runtime
pipewire and wireplumber are user session services. the bluetooth-audio systemd unit is a system service, so it cannot see the user session by default. the fix was setting xdg_runtime_dir=/run/user/1000 in the unit file. one line. without it, the sink did not exist at boot and the script silently did nothing. silent failure is the worst kind of failure for a system service.
4. servo power
the pca9685 v+ pin must come from an external 5v supply, not the pi’s 5v rail. a 4a lipo with a bms feeds the servos. the pi and the battery share ground, and only ground. this is not a creative decision, it is a hardware requirement that the pi 4b’s 5v pin simply cannot supply enough current for two sg90-class servos under load. we burned an evening on this.
5. ncnn model fit
yolofastestv2 was the only model that held 10 fps on the pi 4b at a usable confidence threshold. we benchmarked yolov5n, mobilenet-ssd, and nanodet before settling on it. throughput on the pi 4b is bound by memory bandwidth, not by flops, so the smallest model is not always the fastest. the only way to know is to benchmark on the actual hardware.
what i learned, in three lines
- treat independent subsystems as independent processes. files, signals, and udp are cheaper than threads. you can restart any one of them without taking the others down, and the failure modes are obvious.
- pick the simplest model that meets the latency budget. yolofastestv2 held 10 fps where yolov5n did not, and the difference was not the architecture, it was the memory footprint.
- ship the system integration before the system integration deadline. five daemons wired together at 3 am is a different problem than five daemons running on separate machines. we did this the right way and it still cost us sleep.
shoutout to kaushal and eshwar. this one needed three people to land.
team HEADLESSMONKEY