No description
  • Rust 78.4%
  • JavaScript 11.3%
  • HTML 5.5%
  • CSS 3%
  • D2 1.6%
  • Other 0.2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Ifrahim Ansari 1f7957d922 feat(dashboard): face enrollment UI, ESP32 serial monitor, richer telemetry
Dashboard
- Add webcam-based face enrollment on the users page: capture frames in the
  browser, post them to a new enrollment handler that invokes vision-service
  with a timeout and serializes concurrent enrollments.
- Add `/api/esp/events` and an in-memory ring buffer (`esp_events.rs`) so the
  serial monitor can replay recent encrypted UART frames; sensor events are
  excluded from access totals.
- Persist ESP32 `EVENT:AUTH:*` / `EVENT:SENSOR:*` frames from the UART link
  into `auth_attempts`, with HELLO heartbeats to survive ESP32 reboots.
- Default HLS segments to 1 s for a lower-latency live view.

Firmware
- Build for the `xtensa-esp32s3-espidf` target with build-std and ldproxy.
- Emit PIR, RFID, keypad and PIN decision telemetry over the encrypted link.
- Treat `ESP_ERR_TIMEOUT` on an empty UART ring buffer as "no data".

Vision
- Validate and upsert enrollments (reject split id/name, replace duplicates)
  and log pipeline decisions.

Docs/diagrams
- Add `docs/encryption.md` (D2 + PNG), `diagrams/system_simple.d2`, and
  update the Pi 5 setup, analytics, dashboard, firmware and vision READMEs.
2026-11-10 12:21:08 -05:00
crates feat(dashboard): face enrollment UI, ESP32 serial monitor, richer telemetry 2026-11-10 12:21:08 -05:00
data feat: add Pi vision, liveness and encrypted ESP32 link 2026-11-05 12:41:39 -05:00
diagrams feat(dashboard): face enrollment UI, ESP32 serial monitor, richer telemetry 2026-11-10 12:21:08 -05:00
docs feat(dashboard): face enrollment UI, ESP32 serial monitor, richer telemetry 2026-11-10 12:21:08 -05:00
.gitignore feat(dashboard): face enrollment UI, ESP32 serial monitor, richer telemetry 2026-11-10 12:21:08 -05:00
AGENTS.md feat(dashboard): face enrollment UI, ESP32 serial monitor, richer telemetry 2026-11-10 12:21:08 -05:00
Cargo.lock feat(dashboard): face enrollment UI, ESP32 serial monitor, richer telemetry 2026-11-10 12:21:08 -05:00
Cargo.toml feat: add Pi vision, liveness and encrypted ESP32 link 2026-11-05 12:41:39 -05:00
README.md feat(dashboard): face enrollment UI, ESP32 serial monitor, richer telemetry 2026-11-10 12:21:08 -05:00
shell.nix chore: scaffold Rust workspace and preserve Flask reference 2026-11-05 11:51:21 -05:00

smarty

Smarty is a small physical access-control system for a door:

  • A Rust dashboard (crates/dashboard) with an admin UI, SQLite audit log, camera HLS proxy, lock API, device auth API, and a live vision overlay.
  • An ESP32-S3 firmware (crates/firmware) for PIR, RFID, keypad, servo, I2C LCD, UART-to-Pi and a task watchdog.
  • A Pi vision service (crates/vision-service) that runs face detection, passive anti-spoofing, active liveness and recognition on the camera feed.
  • A Raspberry Pi 5 setup that runs the dashboard and publishes the Camera Module as an RTSP/HLS feed.
Browser ──HTTP──> dashboard (Axum, one process)
                    ├─ SQLite          data/door.db
                    ├─ ffmpeg (1x)     RTSP -> data/hls (last 10 min)
                    ├─ SSE             live face boxes / scores
                    └─ /dev/serial0 ──> ESP32-S3
                                          ├─ PIR / RFID / keypad
                                          ├─ servo lock
                                          └─ I2C LCD

Pi 5 Camera ──RTSP──> dashboard HLS
             └──────> vision-service
                        ├─ SCRFD detection
                        ├─ MiniFASNet passive anti-spoofing
                        ├─ active head-movement liveness
                        └─ ArcFace recognition
                              │
                              └──POST /api/vision/result──> dashboard

Workspace

Path Purpose
crates/dashboard Axum web app: admin UI, lock API, device auth API, camera HLS proxy, live vision overlay
crates/vision Face detection, anti-spoofing, active liveness, recognition and enrollment
crates/vision-protocol Shared live-event types between the Pi service and dashboard
crates/vision-service Pi camera capture + vision pipeline + dashboard client
crates/uart-protocol Encrypted, authenticated Pi ↔ ESP32 UART framing
crates/firmware ESP32-S3 firmware (separate Cargo workspace, Xtensa target)
docs/pi5-setup.md Pi 5 OS, camera, UART, vision, deployment and verification
cad/ Reserved for OpenSCAD sources (empty for now)
data/ Runtime SQLite, HLS and vision models/enrollments (git-ignored)
diagrams/ D2 hardware, power and software diagrams
AGENTS.md Repository conventions and commands for contributors/agents

The Rust dashboard/firmware are the source of truth. The firmware is deliberately a separate workspace because it builds for xtensa-esp32s3-none-elf. The original Python dashboard and Arduino sketches have been retired.

Quick start (dashboard)

nix-shell
cargo run -p smarty-dashboard -- init-db
cargo run -p smarty-dashboard -- create-admin --username admin
cargo run -p smarty-dashboard
# open http://localhost:8000

Development without a Pi works: GPIO/serial/ffmpeg failures fall back to no-ops, and CAMERA_ENABLED=0 disables the camera.

Vision / liveness

The Pi service posts a live VisionEvent for every processed frame. The dashboard broadcasts it over SSE to the camera page, which draws the face box, identity, similarity, passive anti-spoofing result and active-liveness progress on top of the HLS video.

An unlock requires all of:

  1. Exactly one detected face.
  • Passive anti-spoofing passes (MiniFASNetV2).
  • Active liveness passes (the user turns their head slightly).
  • Face recognition matches enrollments.json.
  • The identity matches an active door_users.face_label in the dashboard.

See crates/vision/README.md and crates/vision-service/README.md.

In-dashboard analytics and tests

The Analytics page (/analytics) shows live door/camera/power status, auth success/failure, spoof and recognition rates, lock reliability, latency histograms, recent audit rows, and threshold/target checks. It also has click-to-run software tests and an optional labelled-image vision benchmark. See docs/analytics.md.

An optional /metrics endpoint remains available for Prometheus, but the in-dashboard page does not depend on Prometheus or Grafana.

All dashboard ↔ ESP32 communication is ChaCha20-Poly1305 authenticated encryption with per-frame sessions/sequence numbers. There is no plaintext fallback. Set the same 32-byte key (openssl rand -hex 32) in the dashboard's UART_SHARED_KEY and compile it into the firmware with SMARTY_UART_KEY. See crates/firmware/README.md and docs/pi5-setup.md.

Credential hashing

New administrator passwords and door PINs use salted PBKDF2-HMAC-SHA256 (210,000 iterations, Werkzeug-compatible format). Legacy scrypt hashes are verified and re-hashed on the next successful use. Face embeddings are not hashed because recognition requires the vectors; protect the git-ignored enrollment file instead.

Rolling 10-minute video buffer

The dashboard ffmpeg process keeps the last VIDEO_HISTORY_SECONDS (default 600 seconds) of HLS segments in data/hls. Older segments are deleted by ffmpeg, and any pre-existing segments/playlist are removed at startup. The camera page can seek within the playlist's rolling window.

Firmware

The ESP32-S3 firmware is flashed separately:

cd crates/firmware
cargo build --release
cargo run --release          # espflash flash --monitor

Full prerequisites, wiring, pin map and the UART protocol are in crates/firmware/README.md.

Raspberry Pi 5

The complete Pi 5 guide is docs/pi5-setup.md. It covers:

  • Camera Module → rpicam-vid → mediamtx → RTSP.
  • Pi GPIO14/15 ↔ ESP32 GPIO6/7 UART wiring.
  • Enabling the Pi UART and freeing the serial console.
  • Models, enrollments and the vision service.
  • Systemd units for mediamtx, the camera publisher, the vision service and the dashboard.
  • Verification and troubleshooting.

Configuration

All dashboard settings are environment variables; the complete table is in crates/dashboard/README.md. The most important:

export DATABASE=data/door.db
export HLS_DIR=data/hls
export VIDEO_HISTORY_SECONDS=600
export CAMERA_RTSP_URL='rtsp://127.0.0.1:8554/cam'
export DEVICE_TOKEN='change-me'

# Physical lock output: gpio (Pi relay), uart (ESP32 servo), or none.
export LOCK_TRANSPORT=uart
export SERIAL_PORT=/dev/serial0
export SERIAL_BAUD=115200
export UART_SHARED_KEY='<same 64-hex key as the firmware>'

# Face-triggered unlock cooldown.
export VISION_UNLOCK_COOLDOWN_SECONDS=30

# Optional Prometheus token; not needed for the in-dashboard analytics.
export METRICS_TOKEN=''

Never commit .env files, RTSP credentials, data/*.db, data/vision/*, or HLS output.

Device / firmware API

curl -X POST http://localhost:8000/api/auth/pin  -H 'Content-Type: application/json' -d '{"pin":"1234"}'
curl -X POST http://localhost:8000/api/auth/rfid -H 'Content-Type: application/json' -d '{"uid":"A1B2C3D4"}'
curl -X POST http://localhost:8000/api/auth/face -H 'Content-Type: application/json' -d '{"label":"jane_doe"}'

The vision service uses the stricter POST /api/vision/result endpoint, which also carries passive anti-spoofing and active-liveness results.

Deployment

nix-shell --run 'cargo build --release -p smarty-dashboard -p vision-service'
sudo cp crates/dashboard/smarty-dashboard.service /etc/systemd/system/
sudo cp crates/vision-service/smarty-vision.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now smarty-dashboard smarty-vision

The units assume /home/pi/smarty, keep secrets in /etc/smarty.env, and use a single process for the dashboard because the lock transport and ffmpeg child are process-wide singletons.

Diagrams and CAD

The diagrams/ directory contains D2 sources and rendered PNGs for the power and hardware design. Re-render with

d2 diagrams/<name>.d2 diagrams/<name>.png

cad/ is reserved for OpenSCAD sources. It is intentionally empty for now.