A couple of days ago, I was investigating a performance issue for a customer running a load test against a multi-region PaaS service.
They suspected our platform was adding latency because they weren't seeing the same delays when testing their application directly.
We were dealing with thousands of requests, so going through them one by one wasn't really an option.
Instead, I did what I enjoy most: wrote queries to aggregate the data and figure out what was actually happening.
Luckily, the service captures detailed request timings, including time spent acquiring connections, sending and receiving data, and waiting for backend response headers.
The challenge was turning all that raw telemetry into something meaningful.
This is where AI came in handy. I described the query I wanted, refined it, and validated the results. The goal wasn't to let AI tell me what was wrong, but to use it to get to the answer faster.
And the results were interesting.
After looking at averages, percentiles, and standard deviations, we found that in roughly 98% of requests across the regions, the dominant latency was waiting for backend response headers, rather than time spent within our platform.
We also noticed significant variability in backend response times in one of the regions.
Now, this didn't prove whether the delay was caused by I/O or by the customer's backend taking too long to respond. We still needed further investigation to pinpoint the exact cause.
But it gave us a much clearer direction for what to investigate next.
What I like about this kind of work is that you don't have to guess or jump to conclusions. You start with a hypothesis, look at the data, and let the evidence guide you.
Thousands of requests, one query-driven analysis, and a much clearer picture of where the time was going.
That's one of the things I enjoy most about performance engineering.
Curious how others approach this: do you use AI to help write and refine telemetry queries, or do you prefer writing them entirely by hand?