Asyncio ContextVar Memory Leak: How Sentry's Dedupe Integration Chained Every Failed Scheduler Tick

An asyncio contextvar memory leak is the kind of production bug that sits two layers below where you would first look for it. The scheduler on our staging host had grown to 5.4 GB of resident memory before we noticed. The log line that would have given it away was buried inside a Sentry integration that is on by default. The last four git log entries on the file were unrelated. The symptom was slow; the cause was invisible; the fix was a single call wrapper.

This is the writeup of the leak we chased down inside the automation scheduler behind Wield — our Swiss recruiting-intelligence product, built on a Python backend that runs cron-like agent ticks every minute — and the general lesson that came out of it: a long-lived coroutine loop is not a safe place to live for anything that touches Python's contextvars, and awaiting a coroutine directly from such a loop is a quietly wrong default for anything that can fail.

The Symptom: 0.75 MB per Tick, Reproducibly

The scheduler ticks every minute. On each tick it reads a batch of due automation jobs out of Postgres, enqueues them, and goes back to sleep. On a healthy day it does nothing visible — a tick runs in a few milliseconds and the only trace is a counter on the Prometheus endpoint. On the day we noticed, the counter was still ticking at exactly one per minute, every job was enqueuing fine, every worker was processing fine, and the scheduler's resident set size was 5.4 GB.

Restarting the process dropped memory to 90 MB. One hour later it was at 134 MB. One day later it was back above a gigabyte. The growth was linear, reproducible, and had the exact shape of a leak that scales with the loop iteration count, not with request volume or data volume. On staging the slope was 942 KB per tick — a number we only had because tracemalloc.take_snapshot into our telemetry pipeline gave us a diff per minute. On the production host, shared with Acurio, the slope was lower but non-zero, and over several redeploy-free weeks it was still the dominant memory story on the box.

The jobs themselves were fine. Workers were restarting cleanly, DB sessions were closing, the agent SDK was releasing its HTTP connections. The leak was in the scheduler alone, and it was the scheduler loop itself — not any job it enqueued — that was holding onto something.

The Hypothesis We Ruled Out First

The scheduler does one call per tick that would be the obvious suspect: it opens a SQLAlchemy session, reads a few rows, and closes it in a finally. The first instinct — and the first thirty minutes of the investigation — was that the session was being retained somewhere, probably through a lingering reference on a model instance that had not been detached. objgraph.show_backrefs on the top retainer disagreed. The session was closing. The row objects were going away on the next GC pass. The thing accumulating was not SQLAlchemy state.

The next hypothesis was that our logging stack was buffering formatted messages. We log to stdout for Loki and the scheduler is talkative. logging.getLogger().handlers[0].stream was pointed at a plain sys.stderr; no buffering, no in-memory handler. Nothing in standard logging retains past records.

The third hypothesis was the right one, and it took a stroke of luck in the form of tracemalloc to find: the top retained objects on each diff were TracebackType and the frame objects they transitively kept alive. The scheduler was accumulating exception tracebacks, one per failed tick. Which was odd, because the scheduler was supposed to be swallowing exceptions and continuing.

The Real Cause: Sentry's Dedupe Integration, in the Wrong Context

Sentry's Python SDK ships a default integration called DedupeIntegration. Its job is modest and useful: it suppresses duplicate reports when the same exception bubbles through two different capture points in one request. The way it does it is to keep a reference to the last captured exception in a ContextVar, and to drop the next report if the exception object (or chain) is the same. The reference is per-context, which is the right unit for a web request: each incoming request runs in its own asyncio.Task with its own copy of the context, so one request's deduped exception never bleeds into another's.

The scheduler, though, is not a web request. It is a single while not stop.is_set() loop running inside the main task. Every tick runs in the same context as every other tick. When one tick raised — one of our Microsoft 365 agents was failing on every tick with a 401 from a token that had drifted out of sync, which is a story for a different week — Sentry captured the exception and wrote the exception object into its dedupe ContextVar. The next tick saw that ContextVar still populated. Which, on its own, would have been a small bounded cost: one exception object, replaced each tick, nothing to see.

The leak came from a quirk of what that stored exception transitively holds. For built-in Exception subclasses, the reference is strong, not weak — the dedupe integration cannot use a WeakValueDictionary here because many BaseException subclasses do not support weak references. A strong reference to the exception keeps its __traceback__ alive. The traceback keeps every frame on the failure path alive. Every frame keeps its locals alive. For the M365 tick that was failing, the live locals included the httpx.Response object, which kept the httpx.AsyncClient, which kept the TLS context, which kept the SSL session cache, which kept the connection pool entries. The DB session that opened the tick was in there too. On the order of 900 KB per failed tick, held transitively through one ContextVar entry.

That still would not have been a leak if the ContextVar entry had been replaced cleanly on the next tick. The ingredient that turned a bounded retention into a growing one was asyncio.to_thread. The scheduler offloads the actual tick body to a worker thread — SQLAlchemy's blocking driver is fine on a thread, awkward on the main loop — and asyncio.to_thread copies the current context so that contextvars.copy_context() can be used to propagate context to the thread's frame. That copy is a snapshot. It holds a reference to whatever was in the ContextVar at the moment of the copy. The next tick's failure walked into a context that already held the previous tick's exception, captured a new one, and now the new exception's traceback was walking a frame whose locals included the copied context from the previous tick, which held the previous tick's exception, which held its traceback, which held the frame from the tick before that, which held a copy of the context from the tick before that, and so on down the chain. Every failed tick added one link. The scheduler was leaking the entire history of failures through a chain of context copies hiding inside exception tracebacks.

The reason this was so hard to see from the outside is that every link in the chain looked like one legitimate reference. Sentry held one exception. The exception held one traceback. The traceback held one frame. The frame held one context. The context held one ContextVar with one entry. The entry was one exception. There was no cycle, no suspicious container, no obvious retention — just a chain of singly-held strong references that happened to loop back on itself through the shared loop context.

The Fix: A Fresh Task Per Iteration

The one-line fix is to run each tick in its own asyncio.Task:

# Before
n = await tick(db=db)

# After — fresh context copy per tick
n = await asyncio.create_task(tick(db=db))

asyncio.create_task creates a task with its own copy of the current context via contextvars.copy_context. The task's ContextVar writes do not propagate back into the parent loop's context. When the task finishes, its context is dropped. The dedupe ContextVar for that tick dies with the task that owned it. The next tick starts from the loop's untouched context, which never held any exception reference to begin with. Reproduced in the staging image: 942 KB per tick of growth before, flat after. The same fix went into the worker loop with the same two-line reasoning.

Three things to notice about that fix.

It is not about Sentry. Any library that writes to a ContextVar from an exception handler will behave identically. OpenTelemetry's baggage would. A homegrown "last error" cache would. The leak is in the shape of the loop, not in Sentry's integration. The dedupe integration is doing exactly what it says on the tin; the loop is giving it the wrong lifetime to be correct inside.

It is not about to_thread. We only noticed the chain because to_thread makes the context copy visible — the thread holds a reference to the snapshot while it runs, which gives the garbage collector something to anchor the chain to. The same leak exists, more slowly and more intermittently, in a loop that awaits a plain coroutine directly, because that coroutine's frame also holds the current context implicitly. create_task is the fix for both shapes.

It costs one Task object per iteration. On a scheduler tick that is a few hundred bytes against a tick body that reads from Postgres. On a hot inner loop it would be the wrong trade. The rule we came out with: wrap the body of any long-lived loop that can call into third-party code in asyncio.create_task by default, and only unwrap it when you have measured the task cost and shown it matters.

The General Lesson: Long-Lived Coroutines Are Not Request Contexts

The deeper point is that Python's async stack has two very different notions of context lifetime and they look the same from the inside. The first — a web request, a scheduled job, a short-lived task — lives inside a Task and its context dies when the task finishes. The second — a scheduler loop, a worker loop, a connection reader — lives in whatever task created it, which is often the main task, and its context lives for the lifetime of the process. Any library that was written assuming the first lifetime will quietly misbehave under the second.

Our scheduler was not the first place this bit us. The same shape — long-lived loop, library that uses ContextVar, slow leak invisible to our own code — is why we moved to a dedicated job-table sweep in the pattern we described in the Postgres job-row backstop sweep writeup: a loop that owns its own context boundary per iteration, instead of inheriting one from the process. And it is the same instinct that made us split the Next.js worker in the worker-split with pg_notify piece: give each unit of work a clearly-bounded scope and nothing leaks across.

The investigation itself has a lesson too, which is that tracemalloc snapshots diffed across a long interval are a cheaper first debugger than objgraph, especially for leaks that scale with iteration count. Thirty seconds of tracemalloc.take_snapshot() → sleep → take_snapshot().compare_to(previous, "lineno") would have pointed at TracebackType on the first run, before we spent half a day on SQLAlchemy. Next time we will reach for it first.

If you are running a scheduler, a worker loop, or any long-lived async def on top of a web observability stack, assume at least one of your installed integrations is writing to a ContextVar under the wrong assumption about lifetime, and wrap your tick body in asyncio.create_task as a cheap insurance policy against the next one. The same discipline underlies the per-user analysis queue and the serverless cold-start warmup ping: give each unit of work a context boundary that matches its real lifetime, not the lifetime of the process it happens to live inside.

If you are tracking a slow leak on a Python backend and the top retainers in your tracemalloc diff are TracebackType and FrameType objects you cannot explain, book a free AI Potenzial-Check — we have debugged this exact shape in production more than once.

Sources: Python docs — contextvars · Python docs — asyncio.create_task · Sentry — Default Python integrations