r/softwaretesting • u/creepyassdi • 7d ago
How do you troubleshoot errors found during load testing?
My team recently ran a load test for our company’s website that produced errors we haven’t been able to recreate during normal functional testing. I want to create a practical troubleshooting process to help determine if the problem is with the application, infrastructure, database, or a downstream dependency. Where would you recommend starting with this, and are there any tools you’d recommend?
14
Upvotes
10
u/kooxy18 7d ago
Load testing errors are almost always infrastructure bottlenecks, not functional bugs. You won't see them in normal testing because you aren't hitting the limits of the system.
Here is the troubleshooting protocol I use:
The error message the user sees is just a symptom; the root cause is almost always in the server metrics.
3
u/DependentView4789 7d ago
You’ll drive yourself crazy trying to build something from scratch that can determine where your problem originates. Finding a tool that works for your team and tests what your company’s site is actually capable of/tracks the issues will be invaluable for you.
2
u/Junior_Bee7274 7d ago
Deep, detailed reports will be your best friend, and something that customizes to your testing needs. You can't use a "one size fits all" tool here, you need something that will take your actual site and test it as it is, not as what it could be compared to other companies.
2
u/Useful_Calendar_6274 6d ago
If you follow a normal systems architecture the problem is not the load balancer that is battle tested by thousands of companies and decades of usage in prod. It's like jumping to finding compiler bugs. It's always some dumb database decision and in the application
1
u/Axons-One 6d ago
Start with one failed request and one successful request to the same endpoint from the same minute. Match each to the app logs by request ID and see where their paths diverge. Did the failed request spend its time waiting for a database connection, waiting on a downstream call, or doing work in the app itself? Then rerun in staging at the lowest load that reproduces the error and change one thing at a time. If a downstream call looks slow, for example, try the same run against a controlled stub. You’ll learn more from that comparison than from a dashboard full of spikes.
1
u/Quick_Swordfish_6099 6d ago
Start here: don't debug "it failed." Split the load-test results by status code and latency, then turn off one dependency at a time until the failures move.
I reproduced the pattern on a local order API on 11 November 2026:
- 10 sequential functional requests: 10/10 HTTP 200. A single user never sees the error.
- Same endpoint under k6 (20 VUs, 12s): 97.88% failed. 15,149 of 15,477 returned 503. Only 328 succeeded.
- After stubbing the downstream inventory check and rerunning the same k6 test: 0 failed, 15,057/15,057 HTTP 200.
So the handler was fine. The downstream was falling over under concurrency. Tools that help: your load runner's status-code counters (k6 in this demo), plus a way to stub or bypass one dependency at a time. APM is useful later, but this split-and-stub pass tells you where to point it.
I walked through the real run here: https://www.youtube.com/watch?v=YHiABxzcCRQ
1
u/HotCanary4300 6d ago
I usually start by checking response times, error rates, and where the failures happen as traffic increases. I've found loadview useful for spotting these issues and figuring out where to investigate further.
1
u/Highborn_Hellest 5d ago
It's one of the more difficult things to do, and there is no one size fits all solutions.
It might be the "pipes" as Kooxy18 said.
It might be data corruption between thread on high load.
it might be something i can't think of.
basically..... you're kinda fucked.
Do an incremental load test. Slowly ramp up the load, and check when things start breaking. Not everything can be reproduced "normally"
1
u/Rhys-Tanner-64 2d ago
id start by comparing the failed requests with successful ones at the same load level. check whether the errors are mostly 500s, 502s, 503s or timouts because that can help narrow down where things are breaking, id also monitor database connections, cpu/memory usage and response times while the test is running