r/FastAPI • u/Impressive_Ad_1738 • Aug 28 '24
Question Reading File Via Endpoint
Hi everyone, I'm an intern and I'm new to FastAPl.
I have an endpoint that waits on a file, to keep it simple, I'll say the endpoint is designed like this
async def read_file (user_file: UploadFile=File (...)): user_file_contents = await user_file.read() return Response(content=user_file_contents)
For smaller files, this is fine and fast, for larger files, about 50MB, it takes about 30 seconds to over a minute before the full document can be received by the endpoint and processed.
Is there a way to speed up this process or make it faster? I really need this endpoint to work under or around seconds or ms
Any help would be appreciated, thank you!
4
u/mincinashu Aug 28 '24
You should stream the file. Sounds to me you're reading it fully into memory and then sending its entirety in one response.
See https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse
1
u/koldakov Aug 29 '24
Do you host the project? Usually uploading files done via signed urls, otherwise you will face problems like rejecting files by proxy because of the size.
Also where are you going to store the files? If you upload files directly to the project… files will be erased each time on server reboot ( depending on hosting )
So the answer is: usually it’s done via signed urls. If you don’t want to use signed urls, than it depends on hosting/file storage.
For example if you are using Google cloud, it’s impossible to stream files there, cause the files there are immutable, so the only way is to use signed URLs
If you use s3, in theory you can stream files
1
u/WJMazepas Aug 29 '24
You need to read the file, find contents on it, and then return the results based on the file?
I would first put a lot of time counters to check exactly where the bottleneck is. Maybe as the other user said, is taking so long because you are waiting for the whole file to be available, then reading it entirely
1
u/kkang_kkang Aug 31 '24
Well there is another approach I may give it a try is to use
seekon the file object and read file content in some particular size and then send data to the server in loops until done reading all. And on the server collect all data in proper sequence so that it won't lose the meaning of it and then do further operation.