Async task offloading FastAPI · ARQ workers · Redis broker

The core lesson of the project — keep the API sub-millisecond by moving resource-heavy work off the request path onto independent worker processes.

• Designed a decoupled architecture where FastAPI only enqueues jobs and the actual work runs in a separate, independently-deployable ARQ worker process — the two never talk directly, only through Redis.
• Understood and avoided the event-loop trap: using await asyncio.sleep / non-blocking I/O inside tasks so one slow job never freezes the worker's other concurrent jobs.
• Built for horizontal scaling — throughput grows by spinning up more worker containers reading the same queue, with zero changes to the API.
• Explored Redis internals directly (redis-cli monitor, sorted-set-backed queue, TTLs) to see exactly how the queue, results, and in-progress markers are stored.

Real-time progress streaming WebSockets · Redis Pub/Sub · state vs. event

Replacing wasteful polling with an event-driven pipeline that pushes updates to the UI the instant a worker reports progress.

• Streamed live task progress to the browser over a WebSocket, iterating from a naive server-side poll to a true event-driven design where workers PUBLISH and the socket handler awaits pushes — cutting per-connection Redis reads from ~60 down to one over a 30-second job.
• Handled the two-hop path correctly: Redis Pub/Sub between backend processes (worker → API), and WebSocket from the API out to the browser — Redis never exposed to the client.
• Solved the late-joiner problem by pairing ephemeral events (Pub/Sub) with durable state (a Redis SET key): a browser connecting mid-job reads the current value first, then subscribes for future changes.

Durable persistence & task lifecycle PostgreSQL · async SQLAlchemy

A clean split of responsibilities: Redis holds "what's happening now" (ephemeral); PostgreSQL holds "what happened" (permanent).

• Modelled a tasks table with the async SQLAlchemy ORM and drove each row through its full lifecycle — queued → running → success — with timestamps that make run-duration and stuck/orphaned tasks visible.
• Used the ARQ job id as the shared primary key, linking the Redis queue, the live progress key, and the Postgres record into one identity across all three stores.
• Managed connections the production way — a single pooled Redis connection and DB engine set up once via FastAPI's lifespan, plus history exposed through a read-only GET /tasks endpoint.

What this project taught me Distributed state · Failure modes

• Distributed state management — keeping the browser, the API, and a decoupled worker all agreeing on the status of a single job, across three data stores that each forget or remember on purpose.
• Real operational gotchas debugged first-hand: workers running stale code (no auto-reload), Docker volume-vs-env credential init, and a native-vs-container Postgres port collision.
• When to reach for Redis vs. a relational DB, and why real systems run both — Redis as the fast, forgetful nervous system; Postgres as durable memory.