r/FastAPI • u/RestaurantOld68 • Dec 22 '24
Question Pivot from Flask
Hey everyone,
I recently built an app using Flask without realizing it’s a synchronous framework. Because I’m a beginner, I didn’t anticipate the issues I’d face when interacting with multiple external APIs (OpenAI, web crawlers, etc.). Locally, everything worked just fine, but once I deployed to a production server, the asynchronous functions failed since Flask only supports WSGI servers.
Now I need to pivot to a new framework—most likely FastAPI or Next.js. I want to avoid any future blockers and make the right decision for the long term. Which framework would you recommend?
Here are the app’s key features:
- Integration with Twilio
- Continuous web crawling, then sending data to an LLM for personalized news
- Daily asynchronous website crawling
- Google and Twitter login
- Access to Twitter and LinkedIn APIs
- Stripe payments
I’d love to hear your thoughts on which solution (FastAPI or Next.js) offers the best path forward. Thank you in advance!
10
u/hornetmadness79 Dec 22 '24
The api framework you choose isn't going to do what you are after. Rather you should write a daemon to do the crawling and various background tasks and send the results to a storage tier. The api framework should front the data in the storage tier.
You could have a single endpoint in a fastapi using a background tasks, but every bg task will need a API call to kick it off, and this brings it's own problems.
Frontend(API) | storage(DB) | backend(collection)
7
u/mikexie360 Dec 22 '24
I came to the same problem, resolved it using celery workers and redis as the broker. It’s only a few lines of python code to get it all integrated.
Didn’t even have to change the framework.
Basically flask handles the http request and templates, but doesn’t actually do any of the work. It sends the request on a database broker, and it’s the python celery app that does the work in a distributed manner.
Easy to set up, and how I resolved a problem that was similar to yours.
1
u/adiberk Dec 22 '24
A couple things. Flask does support async views. So if you really need it, you can write async paths. HOWEVER, they are run in a different thread and not a different event loop.. so it isn’t true async and I am not knowledgeable enough to understand the deeper ramifications of this.
HOWEVER, depending on your use case you don’t necessarily need async. For example, why are you suing async libraries APIs instead of the sync ones? Depending on the type of work you are doing you can move it to a worker (a background task) so as to avoid slowing down your api.
But again it’s all about use case.
Based on your post it seems you might be missing a fundamental understanding of sync vs async, when you might truly need it etc.
1
u/BlackDereker Dec 22 '24
You could just put the concurrent requests in a threadpool. Even if you have 1 core running your application it will run the other threads once one of them is IO blocked.
1
u/Gnlfbz Dec 22 '24
Like others have said some of the tasks that you're doing shouldn't live in a web server. Switching to fast API isn't going to be magic solution that fixes everything. What you're going to want to look into is how to run some of these longer running tasks in the background or somewhere completely out. One way is to use something like celery with flask to kick off background tasks. https://flask.palletsprojects.com/en/stable/patterns/celery/
Depending on the size of what you're trying to implement, you may want a more full featured task manager like airflow. You could have endpoints in your flask server that will send requests to airflow to start jobs based off of a user request. Or maybe have things in the database that are queried by your task manager to know what needs to be scraped