KISMET
Local-first image-collection CLI — precise queries, zero API keys
Solo build — CLI, async download pipeline, security model, and web UI.
Overview
KISMET turns a plain list of things (e.g. "vintage Japanese motorcycles," broken into models) into a curated, organized folder of images. Layered category specs let you set a property once on a category (e.g. "12oz can") and every item inherits it. The full folder tree and every search query are shown upfront before any download happens.
Local-first by design: no account, no API key, nothing leaves the machine. Six image providers tried in order with automatic failover, concurrent async downloads, perceptual near-duplicate filtering, and every file is verified and re-encoded in memory before being written — so corrupt files, disguised executables, and embedded payloads never survive the pipeline. A terminal wizard and a FastAPI web UI drive the same engine.
Architecture
Wizard/Web UI (Typer, FastAPI + WS) sits over the engine, with dHash de-dup and resize/crop/rembg. 100% local, zero API keys, nothing leaves the machine.
Technical Decisions
Human-authored layered specs, not a model
Deterministic query building — no LLM guessing — so runs are reproducible, folder names can't be hallucinated, no key/model/network dependency to plan a collection.
Six providers behind one MultiProvider, ordered failover
DuckDuckGo, Openverse, Wikimedia, Unsplash, Pexels, Pixabay share one interface; tries them in configurable order, only fails if every source fails. Keyed providers with no key are skipped, not errored.
asyncio + httpx, up to 12 in flight
Downloading is I/O-bound, so async beats a thread pool — high concurrency, live per-category progress, no GIL contention.
Re-encode every image from verified pixels
Pillow parses structurally in memory; clean pixels re-encoded fresh to disk. Raw response bytes never written — defeats disguised executables and embedded payloads.
Perceptual dHash, not exact hashing
Catches re-compressed/resized near-duplicates that byte-identical hashing misses, with a tunable threshold.
Runs fully offline, keyless by default
No account or credentials needed by default; the 110-test suite runs with no internet; keyed providers are optional accelerators only.
Engineering Challenges
Precise collections without an AI in the loop
An earlier version used a local LLM to brainstorm categories — non-deterministic, tied to a model being installed, could invent folder names that didn't match intent.
Fully human-authored layered category specs — a spec set once cascades to every child item; full folder tree and search queries shown before anything downloads. Deterministic, reproducible, no model/key/network needed to plan.
Trusting nothing that comes off the open web
Images pulled from arbitrary public URLs across six providers of varying reliability — any input (hostile URL, disguised file, path traversal attempt) is untrusted by default.
Zero-trust gate before writing any file: MIME check → size cap → Pillow structural verification in memory → re-encode fresh pixels (raw bytes never hit disk) → path sanitization against directory traversal.
Discovery that never hard-fails on one flaky source
Provider endpoints are uneven (e.g. DuckDuckGo's unofficial endpoint can change without notice); a single failure shouldn't sink the whole harvest.
All six providers implement one DiscoveryProvider interface; MultiProvider iterates in order, catches failures per-source, only raises if every provider fails. Providers needing an absent key are silently skipped; retries handled centrally.
Outcome
Runs end to end with no account, no API key, no data leaving the machine — a terminal wizard for interactive use and a FastAPI + WebSocket web UI (per-image accept/reject, inline query editing, per-category re-run) over the same engine. The 110-test suite passes fully offline. Dropping the LLM planner made the tool faster to start, fully deterministic, and dependency-free at the exact moment that matters most. Runs from a clone (pip install -e .), with a Dockerfile and a Homebrew formula in the repo.
Joy Curry
Production ordering platform for a Manhattan restaurant