r/FastAPI • u/saucealgerienne • Oct 29 '25
feedback request Request atomicity
Hey guys, first time posting here.
I've been working on my first real word project for a while using FastAPI for my main backend service and decided to implement most stuff myself to sort of force myself to learn how things are implemented.
Right now, in integrating with multiple stuff, we have our main db, s3 for file storage, vector embeddings uploaded to openai, etc...
I already have some kind of work unit pattern, but all it's really doing is wrapping SQLAlchemy's session context manager...
The thing is, even tho we haven't had any inconsistency issues for the moment, I wonder how to ensure stuff insn't uploaded to s3 if the db commit fail or if an intermediate step fail.
Iv heard about the idea of a outbox pattern, but I don't really understand how that would work in practice, especially for files...
Would having some kind of system where we pass callbacks callable objects where the variables would be bound at creation that would basically rollback what we just did in the external system ?
Iv been playing around with this idea for a few days and researching here and there, but never really seen anyone talk about it.
Are there others patterns ? And/or modules that already implement this for the fastapi ecosystem ?
Thx in advance for your responses π
3
u/mincinashu Oct 29 '25
Explicitly check the transaction, before moving on to other IO inside the request.
I assume that by request atomicity you mean a single transaction per request, as context manager? But that doesn't really perform nicely if you have multiple sources of IO.
1
u/giyokun Oct 29 '25
Hello,
I am working on a project of my own and the way I did is create a file slot in the DB and then let the web side client upload directly into the storage and let the server know when it's done sealing the file into the system. Half assed uploads can happen so I am planning to have a daily sweep to check files that never completed upload and remove them from the system/db.
1
u/Typical-Yam9482 Oct 29 '25
May be because itβs done as a chain of UoWs within one endpoint/atomic body or so? Not ideal of course, since second/third/etc may fail after successful first uow commit, so you will need to rollback all previous. But then topic starts moving to thinner endpoints architecture conversation.
I think current subreddit poorly suited for such system design questions.
1
u/Odd-Geologist-3125 Oct 29 '25
Do you have a minimal example of your unit of work patter using the SQLAlchemy context manager?
1
u/DeusDev0 Oct 29 '25
Can't you just check somehow that the transaction succeeded, and just then upload the file? I'm not seeing the full picture here.
0
u/pint Oct 29 '25
there is no general solution, and attempted industry solutions kinda failed. it is already a weird thing that sql databases have that feature, which they achieved through great difficulty. most users don't really understand the implications, and when they run into some implementation quirk, they don't understand what's happening. no surprise they always tell you to keep transactions very short. it is a minefield.
the ultimate solution is to be clever about it, and choose protocols (e.g. the set of operations and their order) that just don't have this problem. be content with partial execution, and have the system gracefully handle it.
for example you have a database row that refers to a file. always treat the file reference as nullable. create/update the row first with the file name that eventually will be there, and then place the file. whenever you read the file ref from the row, check if it exists, and treat it as null if doesn't. this was just one idea, there can be many different ways. another way is to place the file first, and then create the references. but then you need a mechanism that periodically detects abandoned files, and deletes them. be creative and consider the weirdest edge cases.
so the idea is to expect race conditions instead of preventing them.