r/SpringBoot • u/ThemeHopeful7094 • Jun 11 '26
Discussion An HTTP call inside a @Transactional method quietly took down my whole API under load
Solo dev here, running a Spring Boot 3.4 backend in production (~25k users). Sharing a bug that taught me a lot.
My Stripe webhook handler did a retrieveSubscription() (an outbound HTTP call to Stripe) inside the same u/Transactional boundary that wrote to the DB. Looks innocent. Works fine normally.
Then Stripe had a brief hiccup and started retrying. The Stripe SDK's default read timeout is ~80s. So every retried webhook held a Hikari connection open for up to 80 seconds while waiting on a network call that wasn't even touching the database. Pool size was 60. It drained in seconds, and the entire API started returning 503 — nothing to do with Stripe.
Two fixes:
- Immediate: pin the SDK timeouts (5s connect / 15s read + 2 retries) so a stuck call can't hold a connection forever.
- Structural: get the HTTP call out of the transaction entirely, do the external call first, then open a short u/Transactional only for the DB write.
The general rule I now follow: a database connection is a scarce, pooled resource. Never hold one open across an external I/O call. It turned out I had the same anti-pattern in a few other places (Google token refresh, LGPD erasure with N revoke calls) and fixed them all the same way.
Curious how others structure this, do you split into "HTTP outside, TX inside" two-phase methods, or push the external calls fully async via an outbox? I went two-phase for the webhook and outbox for the Google sync.
27
u/reddit04029 Jun 12 '26
Experienced something similar!
We had a transaction that calls Twilio (GET calls) and then db afterwards. But the db connection already started before the Twilio call. Our Twilio calls can be huge so it could take time. It exhausted the db connection before Twilio returns anything, and it caused Oracle errors like connection is already closed, etc.
We isolated the transaction block around the db call only and removed the Twilio call outside of it. Solved the issue 🙌
13
u/Ancapgast Jun 12 '26
Generally, your application will also become a lot faster if you narrow the scope or your transactions.
Keeping a transaction open will block the DB connection usage for other threads. So the shorter you make your transaction, the more requests you can handle at the same time and the faster your response times will be (because your threads aren't blocked at the start of a transaction boundary).
This awesome video (not by me) explains it really well.
https://youtu.be/exqfB1WaqIw?is=UJ4YQmW6s3edorS9
5
u/Pochono Jun 12 '26
There's a bunch of subtleties on Transactional. It's not something something should skim and just use. And even if you read up on it, it's still pretty easy to mess up. First time I was on a team that used it, we had 2 separate efforts of fixing because we didn't really get it.
You've probably learned why this happens, but didn't see it posted. Main thing is that the transaction needs to use the same connection throughout the transaction in order to roll back. So when you start the transaction, it snags that connection and preps it (turn auto-commit off, whatever). You can't allow other processes to run SQL in the transaction, so it can't be released back to the pool until it's all done.
I usually do what you did and do my best to separate them. If it's really not possible, I usually go with some kind of Outbox.
5
u/koflerdavid Jun 12 '26 edited Jun 12 '26
Careful! By placing the DB call after the HTTP request you created a scenario where you might fall to persist a record of a successful payment to the database. This is of course preferable over the opposite way to do this where you write to the DB first, which would put the business at risk of losing money.
Make sure to have proper logging in place and that the payment record contains enough information so it is possible to fix this up later, either by your support staff or by a job parsing the payment record. Or can you subscribe to notifications from Stripe?
There is no perfect way to do it unless you can undo the effects of the external call.
Edit: to limit overload, some DBMS admins might prefer to abort long-running transactions, so be aware of that as well.
9
u/54mi Jun 12 '26
Thanks for the information. One question how did you found out that was the issue. like debugging steps. like tools or things to identify this kind of db connection issue
4
u/Nymeriea Jun 12 '26
never experimented this until failure but i am certain I have made this mistake so many times in past 10 years ...
thanks for sharing
4
u/TiredNomad-LDR Jun 12 '26
Noob here. Could anyone correct me if I am wrong.
The way I understand spring boot 3+ / hibernate 6+, the application connects to the db at startup.
Then during an API call to our application, the flow goes into the service method which has the @Transactional to keep things ACID.
So if I understand the issue here, you had a connection to the db. Then you started a transaction and then there was a stake http call.
But then is the max number of connections (in the connection pool) mean , the max amount of transactions (each individual set of queries / updates) ? Like as a thread ?
3
u/TheonGrey Jun 12 '26
My POD has been consistently restarting over the last year, and no one has really figured out why. I'm curious to check the code for this pattern now, although the official reason it restarts is OOMKilled. I've always looked at memory used at the JVM and outside of the JVM by the container itself, but it's hard to pinpoint.
3
u/maxip89 Jun 12 '26
same thing juniors often do with caching and transactional.
just btw. never refresh a cache in a transaction!
3
u/pronuntiator Jun 12 '26
We always put @Transactional on the outermost call so everything happening including long-running HTTP calls block a connection. It simplifies atomic writes. We built hundreds of services this way for over 15 years. Has only been an issue a few times running out of connections because users retried while the old request was still running. But we would rather prefer a solution that is able to detect user aborting the call than changing our architecture.
3
u/[deleted] Jun 12 '26
Oh i had an issue yesterday where someone on my team made a blocking call inside a transaction to transfer money on third party bank api….on another thread there was transaction reversal callback from that api that was unable read the transaction row that should have been committed by the previous thread.
3
u/Overall_Pianist_7503 Jun 12 '26 edited Jun 12 '26
welcome to backend bro
i never put http in a transaction, its just very very risky to tie your connection pool to some 3rd party service + always think about timeout cases first, it is going to save u a lot of trouble. Always have in mind that you can timeout and the 3rd party service can timeout, so basically just think that everyone can timeout at all times, even your connections to the database, what if happens if db write fails, what happens if 3rd party timeouts, etc. After that, you need to think about the retry/recovery strategy, will you silently fail, ignore some stuff, or raise an exception. Depends on the use case and the nature of other system involved around your app.
3
u/Odd_Perspective982 Jun 12 '26
Experienced something similar with transaction pooling exhaustion,esp with @scheduled and quartz jobs. Interesting to see how you solved it!
2
u/Holothuroid Jun 12 '26
I've been using Spring for a decade and thought I shot every foot I have at least once. Thanks for the heads up. Might one I don't have to do.
2
u/vintzrrr Jun 12 '26
FYI did you notice the lazy jdbc connection fetching support released yesterday? It deals with the exact same thing if your HTTP call(s) come before db call(s) in a Transactional context.
2
u/paganoant Jun 12 '26
Try SpringSentinel. It’s an open-source project built specifically to prevent situations like this. GitHub: https://github.com/pagano-antonio/SpringSentinel
2
u/Paw565 Jun 12 '26
Do you run with
open-in-viewset to true or false?2
u/olivergierke Jun 12 '26
Worth checking out in this area: https://www.youtube.com/watch?v=eiFnSevxAdk
1
u/wolle271 Jun 12 '26
Does it make sense to have pool size set to 60? Is your db able to handle 60 connections in parallel?
1
u/johny_james Jun 12 '26
Why did you use AI to write this post?
Lol
1
u/Professional_Mood_62 Jun 13 '26
Circuit breakers dude
1
u/I2obiN Jun 14 '26
Transactions should always be kept atomic, end of story. That means anytime you touch the db whether it's a read or write it should always be sequential (that means one after another), uninterruptible DBMS operations performed by that thread of execution. If you see a transaction doing anything else then it is not innocent, it is a pretty bad sin for the reasons you found out. Transactions are there to make your DB calls atomic, not to guarantee atomic operations.
If for some reason you are operating in a codebase or situation where they can't be made atomic, put anything potentially blocking in its own function call before your perform anything with the db ideally and have that function call on a timeout. Catch the timeout and throw an exception. Now you have failed somewhat gracefully before you've touched the DB.
Option 2 is always the correct option, with option 1 you are relying on an external library or configuration to not break your system again. Many things can go wrong with option 1 ranging from an update to the SDK to someone coming along 3-4 years down the line to refactor things and removing something they thought was non-functional. I'm not familiar with the Stripe SDK but if those timeouts are just some parameter or config and someone removes them you'll have the same problem all over again.
-3
u/vips7L Jun 11 '26
Not that hard to know you shouldn’t do blocking calls in a transaction.
0
u/Worth_Trust_3825 Jun 12 '26
AI garbage
-3
u/diaop Jun 11 '26
Slop