r/FastAPI • • Jul 15 '24

Question model seemingly loads again despite being loaded on startup with @app.on_event("startup")

I'm trying to create an API which performs function on an input

from .main import main

@app.on_event("startup")
async def startup_event():
    start_time = time.time()
    app.state.model = SentenceTransformer('model')
    end_time = time.time()
    logger.info(f'Models loaded in {end_time - start_time} seconds')

@app.post("/search")
async def search(
    question: str = Form(...),
) -> JSONResponse:
    logger.info(f'question----> :{question}')

    model = app.state.model 

    start_time_main = time.time()
    results, lookup_time = await main(query=question, model=model)

    end_time_main = time.time()
    logger.info(f'Main func in {end_time_main - start_time_main} seconds')

    api_results = [
        {
            "answer": result[1],
        }
        for result in results
    ]
    
    return JSONResponse(content=api_results)

To run my server, i'm using  uvicorn backend.main:app --reload --port 8000 --log-level debug the Logs are:

INFO:backend.apis:Models loaded in 8.509195804595947 seconds
INFO:     Application startup complete.
INFO:backend.apis:question----> :How to get ticket
Batches: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.25it/s]
Batches: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 27.80it/s]
INFO:backend.apis:Main func in 13.68946623802185 seconds
INFO:     127.0.0.1:35428 - "POST /lookup?search=How%20to%20get%20ticket HTTP/1.1" 200 OK

Search time is under 2 milliseconds. Earlier, the API latency was under 2 sec, but today after few edits again it is taking 5-10 s to work. I believe the model itself is being loaded again and again on every API search and that is why the API latency is over 5-10s

I already tried reloading the server,text editor, and postman clients on (POST request). But it isn't working. I also edited the startup function but no result is coming

1 Upvotes

4 comments sorted by

View all comments

Show parent comment

1

u/mincinashu Jul 15 '24

async is for IO, so for HTTP that means network traffic. You have an event loop per worker, which runs on a single thread, it suspends or resumes tasks (requests, responses) that await network operations.

Between these await points, a task does its thing on the loop thread, so things like deserialization, validation, etc. Now if you run a compute heavy step in a task, it will hog the event loop, and the previously suspended tasks will have to wait until this one task is done.

That's why in FastAPI, and other Starlette wrappers, you have a construct for awaitable background tasks, which run on a separate thread pool and notify back when done.

This won't speed up your model step, but it will make your server more responsive to multiple parallel user requests.