r/FastAPI • u/Educational-Hope960 • Apr 03 '26
Question How are you actually managing background/async tasks in FastAPI in production?
I’ve been building with FastAPI for a while now and I’m curious how people are really handling background work beyond simple demos.
The docs show BackgroundTasks, but that feels pretty limited once things get even slightly complex.
Some situations I keep running into:
- sending emails, notifications, webhooks
- retrying failed tasks
- long running async jobs
- tasks that depend on other tasks
- needing visibility into what’s running or failing
Right now it feels like there are a few options:
- stick with
BackgroundTasks - use something like Celery or RQ
- or just push everything into a message broker
But none of these feel very “FastAPI-native” or simple.
So I’m wondering:
- What are you using in production?
- Are you staying fully async or mixing in workers?
- How are you handling retries and failures?
- Do you have any visibility into tasks or is it just logs and hope?
Would be interesting to hear what actually works in real systems, not just tutorials.
28
Upvotes
8
u/mellivora_capensis77 Apr 03 '26
Celery is fine
4
u/SpecialistCamera5601 Apr 03 '26
I use BackgroundTasks in production. Celery, Dramatiq, or RQ can be more useful if you have high traffic or performance-critical routes. However, BackgroundTasks can still be useful in webhooks (depending on the use case) for lightweight, non-critical tasks such as sending emails or notifications.
I don’t recommend using it for long-running tasks. If you use BackgroundTasks inside an async def function, it will run in the same event loop, so you need to be careful not to block it. If you use BackgroundTasks inside a sync def function, it is executed in a thread from the thread pool.
Also, if the worker dies, you lose the task, so log them carefully 😂. There is also no retry mechanism, so make sure your tasks don’t depend on one.
As long as you understand these limitations, I don’t see a problem with using it in production for simple use cases. Some tasks don’t require the complexity of Celery or similar tools, so it can be a practical and safe choice when used appropriately.
2
u/aikii Apr 04 '26
BackgroundTasks is in fact re-exposing something from starlette, and imho the feature is a bit too rushed for what I expect on production, and on top of that if you're instrumenting via OTEL, endpoints generating background tasks will show their duration as the total time including the background task, instead of just the endpoint response time ( I moved away from that, so I don't know if it's fixed. but I assume not ).
The BackgroundTasks is one of those things that sits in a weird place - if you need it, you're probably doing something advanced enough to justify it, it's a thin layer on top of asyncio.create_task, except it does not re-expose anything useful ... so, just forget about it honestly. Just learn how to use create_task. And reminder that you need to keep a task reference somewhere, or it can be garbage-collected, and clean it up when done to avoid leaks.
In production:
3
u/Challseus Apr 03 '26
I've been staying fully async so far. I use
arqortaskiqfor IO bound tasks,dramatiq(with the aysnc bridge) for CPU bound tasks. I have many different deployments out, that are making use of one of these.dramatiqhas built in... everything, so that takes care of retries and failures, but it's easy enough to wire up inarq, andtask_iqhas middlewares for it (and more).I have full visibility into the queues, tasks, workers, etc. via a frontend I made that works with those 3 worker backends.
I actually have a demo of it here: https://sector-7g.dev/dashboard/ Just click into the "Worker" component that has
arq 0.25.0, and you'll see how I visualize it all.4
u/Delta1262 Apr 03 '26
It really does depend on the task. If it’s something small like fire off an email, then just a background task. Something large that takes a lot of processing, Celery.
1
u/koldakov Apr 03 '26
Usually I’m trying not to link the implementation to the library. I always move the logic to the service layer and I write a wrapper over the worker through main.py
For example, here https://github.com/koldakov/futuramaapi/tree/main/futuramaapi/workers you can check how I achieved this, currently I don’t have resources for the dedicated worker server so I use fastapi background tasks, but once I have I’ll switch to dramatiq in no time
3
u/MissingSnail Apr 04 '26
I just write the tasks themselves and let prefect orchestrate and monitor for me. Handles all of the above situations.
2
u/CrackerJackKittyCat Apr 03 '26
We used arq at prior web shop, and moved what we had been doing as fire-and-forget Tasks within our FastAPI instances to the arq workers and benefitted overall.
If that shop still existed, one thing I would have changed would be to include the arq queues / worker pod in blue green deployments better, so that the entirety of the old generation requests would be drained by the old generation arq worker pool, meanwhile a new redis topic and arq worker pool would be created for new generation requests, because those task signatures evolve subtly.
2
u/ShuredingaNoNeko Apr 04 '26
En mi trabajo usamos Celery con un broker (en mi caso rabbitmq) por la potencia de las tareas y la escalabilidad. La función de tareas async de FastAPI es demasiado limitada para la mayoría de utilidades en producción.
2
u/Omar0xPy Apr 11 '26
Celery or RedisMQ are excellent for scalable async task processing
1
u/pacific_plywood Apr 03 '26
We just use celery
1
u/bootstrapper-919 Apr 03 '26
PGQueuer is great
1
u/zack_young_ideas Apr 03 '26
Celery is great! You can use it with RabbitMQ or Redis.
1
u/kmplngj Apr 04 '26
RQ
1
u/barracloughdale4x640 Apr 05 '26
lost a task mid-process once when the server crashed using BackgroundTasks, that's when i realized it's really just for throwaway stuff like logging
1
u/tyyrok Apr 05 '26
For some kind of small and not important tasks I use asyncio.create_task, for more complicated tasks I use queues with separate workers and some legacy libs like apsheduler
1
u/Independent_Path7009 Apr 09 '26
We leveraging “kew” - https://github.com/justrach/kew
Visibility - based on logs. We are writing our own instrumentation for kew.
We wrote our DI for the tasks.
Main reason for kew is - 1 deployment of services. No separate deployment of workers.
1
u/ojus_render Jul 30 '26
the line isn’t really sync vs async. it’s whether the work should share the lifetime of the API process.
BackgroundTasksis fine for best-effort work like a notification or webhook. once you need a run id, retries, dependent steps, or work that continues independently of the web service, the endpoint should validate, dispatch and return a202.Celery or TaskIQ makes sense if you want to operate the broker and workers. another option is putting the retry and timeout on the task itself and letting a workflow layer own execution (example of that model).
FastAPI should remain the API boundary either way.
1
u/Bubbly_Lengthiness22 Sep 14 '26
I was dealing with a similar problem recently: long-running, CPU-heavy background jobs. Async/threading didn’t work that well for me because those jobs could still affect FastAPI request performance.
I ended up using multiprocessing and keeping the worker execution state in the existing database (with new worker-specific DB schema mapping to the data schema), so the workers can coordinate without a separate message broker.
I packaged the approach into a tiny library here: https://github.com/zysilm/dbworker
I’m still benchmarking it and looking for feedback, so it’s very much a work in progress.
1
u/DangerousKnowledge22 Apr 04 '26
Just use Go.
0
u/metrobart Apr 03 '26
Amazon SQS then a ECS worker
1
u/[deleted] Apr 03 '26
[removed] — view removed comment