Next.js Worker Split with pg_notify: How We Made an Analyzer Survive Redeploys

A "healthy" redeploy that quietly kills every in-flight user job is one of the more embarrassing shapes of production bug, because everything looks fine until a customer emails to ask where their report went. In our case the answer was: back on the previous Docker image, along with the process-memory Map that held the uploaded PDF texts. The moment CI swapped the image, the new container's Map was empty; the analysis worker inside the same Node process politely wrote sources_unavailable into the run row and gave up. Nothing crashed, no alarm fired, and the customer had to re-upload.

The fix that landed this week is the sort of thing that reads as boring architecture — extract the analysis pipeline out of Next.js into a standalone Bun worker, wire the two over pg_notify, guard the boot with a Postgres advisory lock, and stop treating in-process memory as if it were durable state. What made it interesting was that we did it without going to a second replica. We wanted a single web container and a single worker container, both talking to one Postgres, and we wanted the whole thing to survive a redeploy of either side without a user noticing. This post is the writeup of what we learned about the Next.js worker split with pg_notify pattern along the way.

The bug and the fix both live inside Acurio, our citation-verification product for academic theses (the repo name is zoterohero). The pipeline reads uploaded thesis and source PDFs, extracts claims, retrieves the relevant chunks via pgvector plus BM25, and asks an LLM to judge each citation. Everything used to run inside the Next.js server process. Everything, in a single-instance deployment on Coolify, meant "in the same Node event loop as the HTTP handlers." The design worked. It also drew a line right through the middle of every long-running request the moment we wanted to ship a hotfix.

Why "One Container" Stopped Meaning "No State Problem"

Our own CLAUDE.md had been carrying a warning about this for months, and we had been ignoring it. Three subsystems kept state in process memory — the uploaded source texts in lib/source-store.ts (a global Map plus a minisearch BM25 index), the rate-limit sliding window, and a small subscription seat cache — with a comment that read, more or less, "these silently break the moment a second replica is added." What the warning didn't mention, and what caught us anyway, is that you don't need a second replica to hit the problem — a redeploy has the same effect. A rolling restart in Docker replaces the running container; the new container starts with an empty Map. From the perspective of a citation-verification job that was halfway through eighty-four claims, "empty Map" is functionally identical to "the other replica has never seen this session."

We had also been running the analysis workers inside the Next.js server process itself. instrumentation.ts boots on server start, applies SQL migrations from lib/migrations/, registers a SIGTERM drain, reconciles orphaned jobs, and starts a 30-second resume sweeper plus a 60-second batch poller. On paper this is elegant: one binary, one deploy target, no second thing to break. In practice it means an HTTP request that lands on a busy worker fights the same event loop for CPU, next build restarts terminate the sweepers mid-tick, and any state the workers put in memory rides on the same TTL as the HTTP process — which is "until the next deploy," and we deploy several times a week.

The failure mode we cared about was small and specific: the sources_unavailable 409 that a run raised when it went to look up its uploaded source texts and found the Map empty. It's not an interesting error class in the abstract; a user can always re-upload. But for a product whose entire value proposition is "hand us a thesis, get back a report an hour later," "please re-upload eighty megabytes of PDFs because we redeployed" is not a message you want to send. That single error, and the class of bugs it represents, is what drove the split.

pg_notify and Advisory Locks: The Boring Wire

The Next.js worker split with pg_notify architecture that landed is deliberately dull. The web process gates all pipeline kickoffs on an ANALYSIS_WORKER_MODE=remote env var; when set, kickoff calls become a single pg_notify('acurio_task_kick', payload) and return. A standalone Bun entrypoint (worker.ts) opens a persistent LISTEN client with reconnect and keepalive, reclaims orphaned rows on boot, and runs the sweeper and batch poller that used to live inside the Next.js instrumentation hook. Both processes read and write the same Postgres. Neither one holds anything durable in memory.

Two details from the actual patch are worth flagging.

The first is a pg_advisory_lock wrapped around the boot DDL. Both containers apply migrations at startup; if they race — a fresh deploy where the web container comes up ten seconds before the worker container, or vice versa — you get two connections trying to CREATE TABLE IF NOT EXISTS and CREATE INDEX CONCURRENTLY on the same schema at the same time. Postgres will happily let them, and the loser will fail with a duplicate-relation error that reads, in the logs, as a corrupt install. The advisory lock (SELECT pg_advisory_lock(<hash>) at the top of the boot function, pg_advisory_unlock in a finally) means whichever container reaches the boot first runs migrations to completion, and the second one waits on the same connection-level lock, then no-ops on IF NOT EXISTS. Two lines of SQL. No coordination service.

The second is the claim protocol on the queue rows themselves. Every long-lived table — job_citations, parse_jobs, parse_items — has the same shape: a state column, a claimed_at, a lease_expires_at, and a worker_id. Claiming is a single SELECT ... FOR UPDATE SKIP LOCKED LIMIT 1 inside a transaction that also writes state = 'running' and lease_expires_at = now() + interval '2 minutes'. A background heartbeat every 30 seconds extends the lease; a stall sweeper every couple of minutes releases any row whose lease expired. FOR UPDATE SKIP LOCKED is the mechanism that makes this safe: concurrent workers claim disjoint rows without a distributed lock, without a broker, and without needing to know how many workers exist. The same pattern shows up in the Postgres skip-locked queue pattern that has been well documented for a decade — we didn't invent anything, we just finally used it.

The pg_notify bus is the fast path; the sweepers are the slow-and-correct path. A dropped NOTIFY (the client wasn't listening, or the notify came in during reconnect) does not lose work — the poller picks the row up on its next tick because the row is still queued. That decoupling is the difference between "durable queue backed by Postgres" and "hopeful message bus"; a lot of pg_notify writeups skip the sweeper and end up debugging quiet stalls forever.

The Source Store Was the Real Hostage

Splitting the process was the easy half. The harder half was the source Map that started this whole thing.

The move was to demote the Map from source of truth to L1 cache, and add a session_sources linkage table in Postgres that mirrors the session → uploaded-document mapping at upload time. The write path is best-effort — parse-sources and resolve-sources upsert the linkage but the request never blocks on it; the read path in analysis-jobs and resolve-sources falls back to Postgres when the Map misses, filters by user_id (which is stricter than the in-process owner check), and rebuilds the BM25 index from the persisted chunks. Unlink only happens at the two explicit user-reset sites, not at the post-job memory cleanup — that decoupling is what makes the cache disposable.

The result is that a redeploy mid-flow, or the 24-hour TTL sweep firing while a user is halfway through a batch, no longer produces sources_unavailable. The next request rebuilds the sources from Postgres and the run continues where it left off. The change is invisible to the user, which is exactly what we wanted.

The parse pipeline itself got the same treatment. /api/parse-sources no longer runs pdfjs extraction and embedding on the web event loop; in remote worker mode (gated on a parse_jobs capability heartbeat, so we don't dispatch to a worker that hasn't shipped yet) it stages the raw upload bytes into parse_jobs / parse_items as BYTEA, notifies the worker, and returns 202 {jobId}. The client polls at 1.5s intervals until terminal. file_bytes is NULLed on every terminal transition — success, failure, stall sweep — so the six-hour blob TTL is a safety net rather than the primary cleanup. Admission caps (three active jobs per user, 120 MB staged bytes per user) keep a runaway upload from starving the queue.

There is a version of this writeup that leans into big words — "event-driven microservices," "reactive backend" — and misses the actual shape of the work. The shape is: stop pretending in-process memory is durable, put queues in the database you already have, and let a small standalone process do the LLM-and-CPU heavy lifting. Everything else follows. If you are running any long-running background work inside a Next.js container on Coolify, Fly, or Railway, the same three moves apply. The NODE_ENV-versus-hostname trap we wrote up in our PostHog host-gate post from June rhymes with this one: the design that was correct when the deploy was a single laptop is not the design that survives a single container that redeploys twice a day. Our earlier note on Acurio's citation-verification workflow for Zotero researchers covers what those runs actually do, and the production-failure playbook for AI agents covers the broader "silent quiet failures are the worst kind" theme.

The takeaway that fits on a sticky note: if you can lose it in a redeploy, it is not state — it is cache. Move the state to Postgres, wire the notify bus, and let the sweepers turn "hopeful" into "durable." The sources_unavailable 409 is gone, the redeploy is boring again, and the customer never noticed. That is the standard we are calibrating to.

If you are pushing a Next.js product to a single-instance Docker deploy and starting to feel the same pressure — LLM work fighting the event loop, redeploys eating in-flight work, an in-process Map that everybody knows about and nobody wants to touch — the split we just did is a well-worn path with cheap parts. Book a free AI Potenzial-Check if you want a second pair of eyes on the topology before you commit to it.