Celery + Playwright: The SIGSEGV You Get From Forking a Live Browser

During a testing pass on a Celery-driven web crawler (crawl4ai on top of Playwright/Chromium), a task that had been running fine started dying with:

WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV) Job: 0.')

No stack trace pointing at application code, no exception from the task itself - just the worker process vanishing mid-task. The crawl logic was correct. The bug was one layer down, in how Celery starts its workers in the first place.

The default pool forks

Celery's default worker pool is prefork. When the worker boots, it forks the main process N times (one per configured concurrency slot) to get N worker processes that can pull tasks off the queue in parallel. This is a good default for most Celery workloads - handlers that do HTTP calls, DB queries, or CPU-bound work in isolated, short-lived children.

It's a bad default for a worker whose task drives a browser. fork() duplicates the calling process's memory space, but Playwright/Chromium isn't a passive library call - once launched, it's a real native, multi-threaded process (or process tree, with Chromium's own multi-process architecture) with open file descriptors, GPU/IPC channels, and internal mutexes live at the moment the fork happens.

Why fork() + threads is the actual problem

fork() has a specific, well-documented gotcha: it only clones the calling thread. Every other thread in the parent - simply stops existing in the child. It isn't paused, isn't cleaned up, isn't given a chance to release anything it was holding. The child's memory is a byte-for-byte copy of the parent's, but only one thread of execution continues in it.

That's fine if the parent is single-threaded, or if none of its other threads were holding a lock at the instant of the fork. It's not fine when a native library keeps background threads for I/O, GC, or IPC and one of them happened to hold a mutex, own a file descriptor, or be mid-write to a shared buffer when fork() fired. The child inherits:

• A copy of that mutex, permanently locked, with no thread left alive to unlock it.
• File descriptors and GPU/IPC handles that point at connections the child never actually established - the parent's Chromium process is still on the other end, not something the forked child owns.
• Whatever half-finished state those vanished threads were sitting in.

The child doesn't crash immediately - fork() itself succeeds, and the copy looks intact from the outside. It crashes the moment something in the child touches that corrupted state: dereferences a pointer whose owning thread never made it into the child, or tries to use a handle the child doesn't actually control. That shows up as signal 11, SIGSEGV - a memory access the OS won't allow - and it's why the error surfaces as WorkerLostError rather than any exception from crawl4ai or Playwright: the process is gone before it can report anything.

This isn't specific to Playwright or even Python. It's the same reason you'll see warnings against forking a process that holds an open SQLite connection, a libcurl handle, or anything else backed by native threads and OS-level resources. fork() was designed for an era of simple, single-threaded Unix processes; it never composed cleanly with threads, and every modern native library that keeps its own thread pool re-triggers the same class of bug.

The fix: don't fork at all

Celery ships a solo pool specifically for this situation - it runs every task in the single main process, sequentially, with no forking whatsoever. The browser instance is created and used in the same process for the worker's entire lifetime; there's no fork boundary for it to be corrupted across.

This project sets it in two places, so it's the default no matter how the worker gets started - a celery -A app.worker.celery_app worker invoked directly still gets solo, not whatever pool Celery would otherwise pick:

# backend/app/worker/celery_app.py
celery_app.conf.update(
    task_serializer="json",
    result_serializer="json",
    accept_content=["json"],
    timezone="UTC",
    enable_utc=True,
    task_track_started=True,
    worker_prefetch_multiplier=1,
    # Playwright/Chromium (used by crawl4ai) crashes in forked processes (SIGSEGV).
    # solo pool runs tasks in the main process without fork().
    worker_pool="solo",
)

# backend/Makefile
worker:
 uv run celery -A app.worker.celery_app worker --loglevel=info --pool=solo

Belt and suspenders is deliberate here: the Makefile flag documents the fix for anyone starting the worker by eye, and conf.update makes it true regardless of the command line, including whatever a deploy script or process manager runs in production.

The knock-on effect: silent duplicate vectors

The SIGSEGV wasn't just a crash-and-restart. The crawl task was decorated with autoretry_for=(Exception,) and max_retries=3 - reasonable for a task that hits the network and can fail transiently. But WorkerLostError isn't a normal exception raised inside the task; it's Celery's own detection that the process running the task disappeared. Celery retries it the same way, which meant a crashing task ran up to four times total: one initial attempt plus three retries.

Every one of those crashes happened inside crawl_site_sync, driving the Playwright/Chromium fetch - which is before the pipeline ever reaches filter_new_chunks, the step that records content hashes in the chunk_hashes dedup table. So the sequence looked like:

1. Attempt 1 crashes mid-crawl. Nothing written anywhere yet - clean failure, as far as any downstream state is concerned.
2. Attempt 2 gets lucky, the crawl completes, and the task proceeds past the crawl step into chunking, embedding, and the Zilliz upsert - inserting a full batch of vectors with no dedup guard, because dedup runs after the crawl, not before it.
3. If the next attempt also crashed after the crawl succeeded (retries aren't guaranteed to fail at the same point), it would insert the same vectors again, with dedup providing no protection because filter_new_chunks hadn't recorded those hashes on the previous attempt's crawl either - each attempt starts the pipeline fresh.

The observed result: the same chunk appearing multiple times in vector search results with identical similarity scores. Nothing about the ingestion code was wrong in isolation - the dedup logic worked exactly as designed for its actual job (not re-embedding content already indexed from a previous crawl). It just had no visibility into "this exact task already tried and failed partway through, earlier in this same retry chain," because that failure happened before the point where any record of the attempt existed.

This is worth calling out on its own because the two bugs read as unrelated at a glance - a worker crash and a data-quality problem in a completely different table. They're not. Once the SIGSEGV stopped happening, retries stopped happening, and the duplicate-vector symptom disappeared with it, with no changes to the dedup code at all. A crash in a step with no persisted side effects yet can still fan out into duplicated writes several steps later, purely through the retry mechanism - the crash itself leaves no trace, but the retry it triggers does.

The tradeoff, named honestly

--pool=solo isn't free. It means zero task concurrency within that worker - two crawl tasks queued at the same time run one after the other, not in parallel, no matter how many CPU cores are available. For this project that's an acceptable trade: crawl tasks are I/O-bound (waiting on page loads, not burning CPU) and infrequent enough that serializing them doesn't create a real backlog.

A system with heavier or more frequent browser-driving workloads would need a different shape: keep solo (or a small pool of single-purpose worker processes, each holding its own browser instance) for tasks that touch Playwright, and run a separate worker service on --pool=prefork for everything else - HTTP calls, DB writes, embedding requests - that has no native browser state to protect. Celery supports exactly this via multiple worker processes reading from different queues; nothing about it required for this project's current scale, but it's the natural next step if crawl volume ever demands concurrency Playwright can't get from solo.

Wrap-up

The mechanism is the important part to internalize, because it generalizes past this stack entirely: fork() only clones the thread that calls it, so forking any process that's holding live threads, mutexes, or native handles - a browser, a database connection, a libcurl handle - risks copying that state into a child with no thread left alive to use it safely. The child doesn't fail at fork time; it fails the instant it touches the corrupted copy, which is why the symptom looks like an unexplained SIGSEGV rather than a clear application error. Switching Celery to --pool=solo sidesteps the whole category by never forking in the first place, at the cost of task concurrency - worth it here, and worth re-evaluating the moment that cost stops being negligible.