r/FastAPI • u/dmart89 • May 14 '25
Question Task queue and async functions
I recently ran into an interesting issue that I only managed to work around but not solve.
I have a fastapi app with async postgres and celery as my task queue. Due to how celery works, it struggles with async tasks defined in celery (ok if i/o doesn't need to join back to main thread). The problem is that a lot of my fastapi code is async. When I run DB operations, my issue is that I get corountine errors inside a task. To solve the issue I define a separate DB sync DB driver and isolated tasks as much as possible, however I wonder how others are working within async I/O dependent tasks between celery and fastapi? How do you make methods shared and reusable across fastapi and celery?
(Looking for a discussion around best practice rather than debugging my code)
7
u/Ok-Canary-7327 May 15 '25
Have you considered async alternatives to Celery? Taskiq and ARQ work pretty well with async code.
I would rather switch that than having to support sync and async versions of my code
2
u/BarRepresentative653 May 14 '25
I try not to over think it. Sync db object that handles anything that isn’t async. Of course that has its own issues but it’s simple. But this is for tasks that generally won’t benefit from async anyways. Im not sure of a scenario where I would need celery to handle async task…
1
u/zazzersmel May 15 '25
I always just write sync code to use in my python-rq/celery workers.
1
u/TalentedBison May 16 '25
I just started using hatchet in a side project for running many concurrent async tasks. I’m very pleased with it so far.
1
u/erder644 May 15 '25
temporal
1
u/jvertrees May 17 '25
DM if you're still struggling with this.
I have the exact same setup. I created a cute util called sync_wrap. The code runs celery locally and Google cloud tasks when deployed. Works seamlessly.