Assistive doorbell with on-device inference

A doorbell whose entire interface is a sound is useless to the person it is meant to serve. This one detects the visitor and puts the alert on a phone, where it can actually be seen.

Built: Dec 2021 - Mar 2022
Hardware: Raspberry Pi + camera
Detector: MobileNet-SSD
Latency: ~470 ms end to end
Alerting: IFTTT webhooks

Frame to alert: camera, on-device detection, confidence threshold, then a push notification and a lamp. No audio path anywhere in it.

01Who it was for

A doorbell is an audio device pretending to be a universal one. If you are deaf or hard of hearing, the entire interface - a chime in another room - conveys nothing. The usual workarounds are a flashing lamp wired to the bell push, which only works if you are in the room with the lamp, or a camera app you have to already be watching.

The design goal was narrower and more useful than "smart doorbell": the alert has to reach the person wherever they are, and it has to fire when someone arrives - not only when they think to press a button. That second half is what makes it a vision problem rather than a wiring problem.

Visual and haptic, never audio. A phone notification is the primary channel, because a phone is already the device that is always within reach.
Detect arrival, not just the button. A visitor who cannot find the bell, or a delivery left at the door, should still raise an alert.
Two-way from the same device. The notification opens an intercom path, so the conversation does not depend on hearing a doorbell in the first place.
Runs on hardware you can actually buy. A Raspberry Pi and a camera module, no cloud subscription and no monthly fee.

02The pipeline, and where the time goes

The whole system is one loop: pull a frame, prepare it, run detection, decide, notify. The interesting engineering is not any individual stage - it is the budget. On a Raspberry Pi the inference is heavy but the network call is heavier, and that ordering determines the whole design.

One visitor, end to end

A single frame walking the pipeline. The bar underneath fills with each stage's real share of the latency budget.

CapturePiCamera

Preprocessresize + norm

InferenceMobileNet-SSD

Thresholdconf > 0.6

NotifyIFTTT webhook

Capture 33 ms

Preprocess 12 ms

Inference 180 ms

Threshold 2 ms

Notify 240 ms

Capture 33 msPreprocess 12 msInference 180 msThreshold 2 msNotify 240 ms

waiting for a visitor…

Someone is at the door Person detected · 0.91 confidence · tap to open the intercom

Roughly 470 ms from photons to phone. Inference is 180 ms of it - but the outbound webhook is 240 ms, and that is a network round trip nothing local can optimise away.

This is why the notification is fired before the intercom session is set up rather than after. The alert is the latency-critical path; the audio channel can take another second to negotiate because by then the person is already looking at their phone.

03Making detection cheap enough

A Pi will not run a large detector at video rate, and it does not need to. Almost every frame at a front door contains nothing at all, so the job is to spend as little as possible establishing that.

gate 1

Motion first

Frame differencing in NumPy over a downscaled greyscale image. It costs almost nothing and rejects the overwhelming majority of frames before the network is ever loaded with one.

gate 2

Detect second

Only frames that survive motion gating reach MobileNet-SSD. The class filter is narrow - person and a few package-shaped objects - because everything else is noise for this use case.

gate 3

Confidence and dwell

A single confident frame is not an arrival. The detection has to persist across consecutive frames, which removes the false positives from a cat, a shadow, or a passing car.

gate 4

Cooldown

After an alert the pipeline suppresses further notifications for a fixed window. Without it, one visitor standing at the door produces a notification per frame.

Those four gates are the difference between a project that demos and one you would actually install. The first working version had none of them and sent about forty notifications for a single delivery.

04What I would do differently

The webhook is the weak point. Routing alerts through IFTTT was the fastest path to a phone notification, but it puts a third-party service on the critical path of an accessibility device. A direct push service, or a local push to devices on the same network, would be both faster and less fragile.
Frame differencing is fooled by light, not just motion. A cloud crossing the sun triggers the whole frame. An adaptive background model would cost a little more per frame and reject far more.
No on-device record of what triggered an alert. When a notification turned out to be wrong there was no stored frame to look at, which made tuning the thresholds much slower than it needed to be.
Accessibility work needs the user in the loop early. Several decisions I made on latency grounds mattered far less than the wording and persistence of the notification itself, which I only learned by watching someone use it.

This was the first thing I built where the correctness criterion was not a benchmark but whether a specific person could rely on it - and it is still the project that most changed how I think about what "working" means.