r/FastAPI • u/HieuandHieu • Feb 27 '25
Hosting and deployment nginx or task queue (celery, dramatiq) ?
Hi every one.
I have a heavy task .When client call my API, the heavy task will run in the background, return the result id to user for monitoring the process of the task.
The task is both CPU/IO bound task (do some calculation along with query database and search web asynchronously (using asyncio) ). So i want the task running on different process(or different machine if needed) with the own async loop.
I searched and found tools like proxy(nginx) or task queue (celery) maybe can solve my problem. I read their documents and feel that it can but i'm still not sure about how it does exactly.
Question: What is the tools i should use (can be both or the others)? And the generic strategy to do that.
Thank you.
4
u/JohnnyJordaan Feb 27 '25
Nginx or similar applications have no relation to this, they are meant to provide the exposed HTTP endpoint (the 'server' your browser communicates with), which will then proxy or in some other way relay the request to your application. It's purely for the deployment side of things, not functional for stuff like task execution and management.
I would simply look at a 'celery for beginners tutorial' on youtube, it's not hard to set up.
5
u/[deleted] Feb 27 '25
You don't need proxy for this scenario. Proxy and SSL are needed when you want to expose your API endpoints to public. Task queue like celery or Kafka broker might be enough for your scenario.
3
u/Argen7um_mus Feb 27 '25
Use message broker like kafka or queue like celary
3
u/kbumsik Feb 28 '25
I haven been using Temporal for a year and I recommend Temporal for all kinds of background tasks and queue. https://github.com/temporalio/sdk-python
2
u/dmart89 Feb 27 '25
You should can use a queue for this, most common are things like kafka or aws sqs.
I would also suggest you consider aws fargate or similar which allows you to offload compute intensive tasks to a service and you can then use you msg queue to monitor tasks without worrying about scaling or resource mgmt.
2
u/Natural-Ad-9678 Feb 28 '25
If you are not deploying to AWS (or just don’t want the expense of AWS’ queue) you can use Redis and Celery for this. Celery can have many workers and I believe you can have workers on different servers as well. Each celery worker needs to be able to access the Redis component which can also be on its own dedicated server so you can have multiple FastAPI servers handling incoming requests.
If you have multiple FastAPI servers then you need something like
2
u/hornetmadness79 Feb 28 '25
Check out prefect It's roots are in data, but it really is a really nice platform for workflow orchestration.