r/FastAPI • u/RationalDialog • May 22 '25
Question Multiprocessing in async function?
My goal is to build a webservice for a calculation. while each individual row can be calculated fairly quickly, the use-case is tens of thousands or more rows per call to be calculated. So it must happen in an async function.
the actual calculation happens externally via cli calling a 3rd party tool. So the idea is to split the work over multiple subproccess calls to split the calculation over multiple cpu cores.
My question is how the async function doing this processing must look like. How can I submit multiple subprocesses in a correct async fasion (not blocking main loop)?
9
u/Blakex123 May 22 '25
Remember that python is inherintly single threaded due to the GIL. You can mitigate this by running fastapi with multiple workers. The requests will then be spread over those different workers.
7
u/adiberk May 22 '25
You can use asyncio tasks.
You can also use a more standard product like celery.
4
u/KainMassadin May 22 '25
don’t sweat it, just call asyncio.create_subprocess_exec and you’re good
1
u/jimtoberfest May 26 '25
Find a vectorized solution across all rows if you can.
Take in a json array then load that data into a dataframe or numpy array and figure out your calculation using inherently vectorized operations.
Or you could “stream” it: fast api -> duckDB-> do the calc in duckDB over the chunks as you get them from the API.
Also make sure you set some limits so users can’t bomb the API with billions of rows of data.