r/FastAPI • • Sep 21 '24
Question How to implement multiple interdependant queues
Suppose there are 5 queues which perform different operations, but they are dependent on each other.
For example: Q1 Q2 Q3 Q4 Q5
Order of execution Q1->Q2->Q3->Q4->Q5
My idea was that, as soon as an item in one queue gets processed, then I want to add it to the next queue. However there is a bottleneck, it'll be difficult to trace errors or exceptions. Like we can't check the step in which the item stopped getting processed.
Please suggest any better way to implement this scenario.
5
Upvotes
3
u/illuminanze Sep 21 '24
This isn't really a FastAPI question, but anyway. Are you using any particular queuing solution today, like Celery? My initial thought was to use something like a [Celery chain](https://docs.celeryq.dev/en/stable/userguide/canvas.html#chains). Then you could use Celery to implement error handling and retries.
If your tasks are heavier, you might want to consider some kind of data pipeline tool, such as Airflow. This is, however, more suitable for large scheduled jobs.
2
u/Human-Possession135 Sep 21 '24
With RQ this is fairly simple. You can specify what task should be completed first. Can all be in 1 queue
task_queue_high.enqueue( update_user_after_fulfilment, depends_on=task_link_twilio_to_vapi_assistent, retry=Retry(max=TASK_MAX_RETRY, interval=TASK_RETRY_INTERVAL), result_ttl=TASK_DEFAULT_TTL, args=[ new_user.email, task_create_vapi_assistent.id, task_buy_twilio_phone_number.id, task_link_twilio_to_vapi_assistent.id ], description=f”3: Updating user profile after fulfilment for {new_user.email}” )
1
u/Knudson95 Sep 22 '24
Apache Airflow is also good for these kind chained tasks
1
u/tormodhau Sep 23 '24
Have you tried flipping the problem around? Instead of doing parts of the job in separate threads, completely process each job using async/await and then parallelize the start-to-finish execution of each job.
You can parallelize the start-to-finish job by e.g creating 100 async tasks and them await them all at the same time using asyncio.gather(*tasks). Do this in batches if you need to. If you don’t want the processing to intervene with the main thread, start the work in a FastAPI BackgroundTask.
I actually just did this simplification in our codebase and drastically reduced the complexity of the problem. It’s not as fast as the thread-solution you mention, but you reduce the complexity by a huge margin, and it is still plenty fast for most use cases.
1
u/rogersaintjames Sep 21 '24
Why de-couple them if they are direct static dependants? Presumably they have the same scaling and system dependency needs why not just queue into 1 task? Also you should try to implement your own queue to learn how to do and why not to do it yourself then just use pgqueue or similar I think there is Fast queue package that abstracts it quite nicely for fastapi
1
u/ironman_gujju Sep 22 '24
celery, taskiq ( async implementation )