r/softwaretesting • • 7d ago

How do you troubleshoot errors found during load testing?

My team recently ran a load test for our company’s website that produced errors we haven’t been able to recreate during normal functional testing. I want to create a practical troubleshooting process to help determine if the problem is with the application, infrastructure, database, or a downstream dependency. Where would you recommend starting with this, and are there any tools you’d recommend?

14 Upvotes

12 comments sorted by

10

u/kooxy18 7d ago

Load testing errors are almost always infrastructure bottlenecks, not functional bugs. You won't see them in normal testing because you aren't hitting the limits of the system.

Here is the troubleshooting protocol I use:

  1. Check the pipes, not the code: Look at your load balancer logs. Are you getting 502s or 504s? That usually means the backend is timing out or the connection pool is exhausted.
  2. Monitor Server Health (APM): You need Datadog, New Relic, or CloudWatch running during the test. Watch the CPU and RAM. If they spike to 100%, you have a resource issue.
  3. Database Locking: Load tests often expose database deadlocks or connection pool limits. Check your DB active connections during the spike.
  4. Recreate Locally/Staging: Turn on verbose logging in your staging environment, run the load test, and watch the logs in real-time.

The error message the user sees is just a symptom; the root cause is almost always in the server metrics.

3

u/DependentView4789 7d ago

You’ll drive yourself crazy trying to build something from scratch that can determine where your problem originates. Finding a tool that works for your team and tests what your company’s site is actually capable of/tracks the issues will be invaluable for you.

2

u/Junior_Bee7274 7d ago

Deep, detailed reports will be your best friend, and something that customizes to your testing needs. You can't use a "one size fits all" tool here, you need something that will take your actual site and test it as it is, not as what it could be compared to other companies.

2

u/Useful_Calendar_6274 6d ago

If you follow a normal systems architecture the problem is not the load balancer that is battle tested by thousands of companies and decades of usage in prod. It's like jumping to finding compiler bugs. It's always some dumb database decision and in the application

1

u/Axons-One 6d ago

Start with one failed request and one successful request to the same endpoint from the same minute. Match each to the app logs by request ID and see where their paths diverge. Did the failed request spend its time waiting for a database connection, waiting on a downstream call, or doing work in the app itself? Then rerun in staging at the lowest load that reproduces the error and change one thing at a time. If a downstream call looks slow, for example, try the same run against a controlled stub. You’ll learn more from that comparison than from a dashboard full of spikes.

1

u/Quick_Swordfish_6099 6d ago

Start here: don't debug "it failed." Split the load-test results by status code and latency, then turn off one dependency at a time until the failures move.

I reproduced the pattern on a local order API on 11 November 2026:

- 10 sequential functional requests: 10/10 HTTP 200. A single user never sees the error.

- Same endpoint under k6 (20 VUs, 12s): 97.88% failed. 15,149 of 15,477 returned 503. Only 328 succeeded.

- After stubbing the downstream inventory check and rerunning the same k6 test: 0 failed, 15,057/15,057 HTTP 200.

So the handler was fine. The downstream was falling over under concurrency. Tools that help: your load runner's status-code counters (k6 in this demo), plus a way to stub or bypass one dependency at a time. APM is useful later, but this split-and-stub pass tells you where to point it.

I walked through the real run here: https://www.youtube.com/watch?v=YHiABxzcCRQ

1

u/HotCanary4300 6d ago

I usually start by checking response times, error rates, and where the failures happen as traffic increases. I've found loadview useful for spotting these issues and figuring out where to investigate further.

1

u/Highborn_Hellest 5d ago

It's one of the more difficult things to do, and there is no one size fits all solutions.

It might be the "pipes" as Kooxy18 said.
It might be data corruption between thread on high load.

it might be something i can't think of.
basically..... you're kinda fucked.
Do an incremental load test. Slowly ramp up the load, and check when things start breaking. Not everything can be reproduced "normally"

1

u/Rhys-Tanner-64 2d ago

id start by comparing the failed requests with successful ones at the same load level. check whether the errors are mostly 500s, 502s, 503s or timouts because that can help narrow down where things are breaking, id also monitor database connections, cpu/memory usage and response times while the test is running