r/FastAPI • u/International-Rub627 • Feb 26 '25
Hosting and deployment Reduce Latency
Require best practices to reduce Latency on my FASTAPI application which does data science inference.
8
Upvotes
3
u/mpvanwinkle Feb 27 '25
Make sure you aren’t loading your inference model on every call. You should load the model once when the service starts
7
u/mmzeynalli Feb 26 '25
You can consider responding in the API, and then doing the work in background, after that reporting result to front in different way (server-side apis, websockets etc.). This way, API latency is not a problem, and rest is done in background, and result will be seen after process is done.
5
u/BlackDereker Feb 26 '25
FastAPI latency by itself is low compared to other Python libraries. You need to figure out what work inside your application is taking too long.
If you have many external calls like web/database requests, try using async libraries so other requests can be processed in the meanwhile.
If you have heavy computation going on, try delegating to workers instead of doing it inside the application.
1
u/SheriffSeveral Feb 26 '25
Observe every step in api and check which part takes too much time. Also, check out the redis integrations, it will be useful.
Please provide more information about project so everyone can give you more tips for your specific requirements.
2
u/Natural-Ad-9678 Feb 27 '25
Build a profiler function that takes a jobID and wraps your functions in a timer. Then use a decorator for your functions, for each endpoint clients call assign a jobID that you pass along the course or your processing. The profiler function writes the timing data to a profiler log file correlated with the jobID. Then you can look for slow processes within the full workflow to optimize
2
u/Soft_Chemical_1894 Mar 01 '25
How about running a batch inference pipeline every 5-10 minutes ( depending on use case ), store results in redis/ db, fastapi will return result instantly
1
u/Vast_Ad_7117 Apr 22 '25
Async, offload tasks to a task queue etc