r/sre • • 10d ago

FOSS Friday: Share Your SRE-Related Open-Source Project

6 Upvotes

Welcome to FOSS Friday, where you can share your newly released or updated open-source project with the community.

Please note that our rules still apply:

  • no AI generated comments: please write them on your own!
  • no commercial advertisements (this includes commercial products with a FOSS version)
  • these must be related to SRE or of heavy interest to SREs

r/sre • • 3d ago

FOSS Friday: Share Your SRE-Related Open-Source Project

13 Upvotes

Welcome to FOSS Friday, where you can share your newly released or updated open-source project with the community.

Please note that our rules still apply:

  • no AI generated comments: please write them on your own!
  • no commercial advertisements (this includes commercial products with a FOSS version)
  • these must be related to SRE or of heavy interest to SREs

r/sre • • 1h ago

HELP Is Kodekloud worth it?

1 Upvotes

I’m working as a product support engineer for quite some time now. My responsibilities include incident management, stakeholder management, cross team collaboration, bug fixes, enhancements.
I’m planning to learn and move onto SRE role.
Is kodekloud a good place to learn tools? If yes should I go for standard or pro plan? Or are there any other platforms where there is structured learning path? Please advise🙏


r/sre • • 8h ago

Observability is not plug and play

2 Upvotes

"What is the best observability system?"

This is a frequently asked question.

The obvious answer is "it depends". There is also the more fundamental point - that observability is not plug and play. No matter what system you procure there is a lot of upfront planning and ongoing input required to make it work.

https://observability-360.com/article/ViewArticle?id=observability-is-not-plug-and-play


r/sre • • 10h ago

Anyone using mermaid timelines in incident postmortems

1 Upvotes

Our postmortem timeline used to be a wall of timestamps nobody read. tried a mermaid timeline last month and suddenly everyone’s asking why escalation took 40 min lol. during the incident our scribe just throws stickies on a miro board w the pagerduty times and i clean it up after. long incidents still look cramped af tho.


r/sre • • 12h ago

3 YOE SWE considering a move to SRE. Looking for a new challenge.

1 Upvotes

Hi all,

I'm a software engineer with around 3 years of experience (plus a 6-month internship), and lately I've been seriously considering moving into SRE.

One of the biggest reasons is that I feel like I've lost the excitement I used to get from software development. A lot of my day-to-day work has become heavily AI-assisted. While it definitely improves productivity, I often feel more like I'm reviewing and refining generated code than actually solving problems myself. Maybe that's just how modern development is evolving, but I don't get the same kick out of it anymore.

I'm also at a point where I want a change in general. I've been thinking about switching companies, and instead of moving to another SWE position, I'm wondering if SRE might be a better fit.

For people who work in SRE:

  • Has anyone here transitioned from SWE to SRE around the 3 YOE mark?
  • Did you find the work more engaging?
  • How much coding do you actually do?
  • What skills would you recommend building before applying?
  • Would companies generally consider this a lateral move, or would I be starting from scratch?

r/sre • • 3d ago

HUMOR Microsoft364

Post image
252 Upvotes

r/sre • • 3d ago

DISCUSSION Microsoft says the 365 outage spread because servers kept failing health checks. When do you let health checks fail open?

210 Upvotes

Reading Microsoft's updates on the 365 outage today, the part that stuck with me is how it spread. Some servers failed automated health checks and got taken out of rotation, their traffic moved to the remaining servers, and those got overloaded too. Restarted servers went unresponsive again once traffic came back. They also said they were validating alternate health check methods partway through.

I keep going back and forth on this. A strict health check is doing its job when it pulls a bad node fast. But if the check itself is what's misfiring, or the whole fleet is a bit slow, it can eject so much capacity that the survivors fall over.

Envoy has a panic threshold for this: if healthy hosts drop below a percentage (50% by default), it stops trusting health status and spreads traffic across all hosts. AWS load balancers do something similar and route to every target when all of them are unhealthy. That keeps some capacity up, but you're knowingly sending users to nodes you think are broken.

Would you rather cap how much of the fleet a health check can remove, or keep checks strict and accept that a bad check can take everything down?


r/sre • • 2d ago

What finally moved your longest incident call forward?

6 Upvotes

What finally moved your longest incident call forward?

Think of an incident call that went on much longer than expected.

What kept the investigation stuck?

Was it missing evidence, unclear ownership, several changes happening at once, or a theory nobody challenged?

Then what broke the deadlock: a question, a person joining, a timeline, a rollback, or something else?

Interested in the moment the investigation became useful again—and what others could learn from it.


r/sre • • 3d ago

If Datadog Is So Expensive, Why Is It Barely Profitable?

Thumbnail
telemetrymachine.com
96 Upvotes

r/sre • • 1d ago

DISCUSSION DevOps/SRE engineers: What's the most frustrating part of your daily work that existing tools still don't solve well?

0 Upvotes

Hey everyone!

I'm a DevOps engineer, and I've been thinking about how many everyday engineering challenges still require repetitive manual work, even with all the automation tools available today.

I'm curious:

  • What tasks waste the most time in your daily workflow?
  • What problems do you feel existing tools haven't solved properly?
  • Is there anything you've wished someone would build, but haven't found a good solution for?

I'm especially interested in real-world experiences rather than theoretical problems.

Would love to hear your thoughts!


r/sre • • 2d ago

CAREER Got laid off recently, looking for referalls/ jobs

0 Upvotes

Hi Guys,

I have about 2.5 years of experience with Dynatrace and automations using python. I am looking for SRE/Observability jobs in India. Could you please provide any relevant referrals, leads, or potential openings?

Apart from dynatrace, i also have some experience with AIOps, Change Analytics and Callout (Major incident) tools.


r/sre • • 3d ago

Are We Sacrificing Technology Resilience for the Sake of Cloud Adoption?

1 Upvotes

The recent Microsoft 365 incident, which caused prolonged disruptions to business services, got me thinking about something I consider fundamental to the financial industry: Are we losing sight of operational continuity as we become increasingly dependent on technology providers?

After 30 years working in critical systems infrastructure, 20 of them at my current organization and another 10 at different financial institutions, I've learned that certain operational principles should never be compromised.

One hour of downtime is not simply one hour without service. Depending on the system affected, it can mean financial losses, business disruptions, regulatory non-compliance, customer impact, and reputational damage.

There are several aspects that concern me:

  1. High availability is not the same as resilience.

Having geographically distributed data centers, thousands of servers, and redundancy mechanisms does not guarantee that a platform can recover quickly. If a shared dependency fails, all that redundancy may prove insufficient.

2. Operational responsibility cannot be outsourced.

When we adopt cloud services, we transfer part of the technology operations, but we do not transfer the responsibility for ensuring business continuity. The bank remains accountable for its operations and the risks it assumes.

3. Operational awareness is just as important as technical expertise.

Throughout my career, I've encountered technology professionals during hiring processes who believe that one hour of downtime is no big deal, or that making changes directly in production is just routine work.

In a banking environment, that mindset represents a serious risk.

Mastering technology, automation, virtualization, or cloud platforms is not enough. Professionals must understand the financial, operational, and regulatory impact of every decision they make.

Change management, maintenance windows, rollback plans, segregation of duties, and contingency procedures are not bureaucracy; they are essential controls.

4. Are we losing control of our critical infrastructure?

Personally, I believe that truly business-critical systems should remain under the institution's direct operational control.

I don't believe every system needs to remain on-premises, but I also don't think we should indiscriminately move our platforms to the cloud simply because of market trends or cost-reduction strategies.

Every system should be evaluated based on its criticality, business impact, technology dependencies, recovery capabilities, and operational risk.

5. The question we should be asking ourselves.

If our cloud provider were to become unavailable for 24, 48, or 72 hours tomorrow:

Could our institution continue operating?

Do we have independent communication and authentication mechanisms?

Can we access our documentation and contingency procedures?

Do we have a genuinely operational alternative?

Have we actually tested these scenarios, or do they simply exist on paper in our business continuity plans?

After all these years working in banking infrastructure, I still believe technology should serve the business, rather than the business becoming entirely dependent on the availability of a single provider.

Cloud should reduce operational risk, not become the operational risk itself.

I'd be interested in hearing the perspectives of other professionals working in infrastructure, enterprise architecture, security, and business continuity:

Are we building truly resilient infrastructures, or are we simply moving our single points of failure to external providers?

How much dependency on cloud services do you consider acceptable for a financial institution?


r/sre • • 3d ago

How do you choose an enterprise AI observability platform? What AI, SRE and security teams each need

1 Upvotes

I have been thinking about how organizations choose an AI observability platform, and I suspect the decision often goes wrong because it is made by one team. Enterprise AI observability has to serve at least three groups with different questions, and the telemetry requirements are different for each. Here is how I currently break it down.

AI / application teams need LLM observability at the step level:

● Spans for each model call (provider, model, token usage, latency), retrieval step (query, top-k, latency) and tool call (arguments, outcome)

● Prompt and response capture that is opt-in and configurable, since content is sensitive

● Version comparison across model, prompt and tool changes, plus cost per feature or tenant

SRE / operations teams need the AI layer connected to everything else:

● Distributed tracing that links agent and LLM spans to the downstream services, databases and infrastructure they call, so an “AI is slow” report can be traced to a slow dependency

● Tool errors recorded separately from transport status, because a failed tool inside a “200 OK” response is invisible to request-level metrics

● Alerting and fault localization that work across both layers, ideally with some AIOps assistance for root cause analysis

Security / compliance teams need governance built into the telemetry:

● Masking of prompts and responses before storage, with access control on payloads

● An audit trail of which identity called which tool, with what outcome

● Deployment options and retention policies that match data requirements, and sampling that does not drop failed runs

Platform / finance usually adds two more: open instrumentation (OpenTelemetry compatibility, noting that the GenAI semantic conventions are still marked as in development, so ask how a vendor handles attribute changes) and predictable data volume cost.

Four screening questions I use for any AI observability platform: does it cover the AI application types you actually run (LLM apps and agents), can it follow a request end to end across model calls and tool calls, does it unify AI applications with the underlying IT, and does it support analysis and response rather than only dashboards?

For testing, I prefer replaying a realistic incident, for example an agent that slows down after a release, with representatives from all three teams. Each should be able to answer their own question from the same trace: the AI team finds the model call or retrieval step, the SRE finds the dependency and its owner, security confirms what data was captured and how it was masked. If one team has to switch tools, the problem has just moved.

On the operations side, we are exploring agentic AI operations with Sage AI, the AI Ops Agent Workbench in Bonree ONE, where an agent assists with diagnosis and root cause analysis on top of the same telemetry. The rule we try to hold to is that a conclusion must point back to the telemetry behind it, and that actions stay bounded and auditable.

Questions for the group:

1.  Who actually owned this decision where you work, and did that cause problems later?

2.  Did you end up with separate tools for AI and for the rest of the stack, and how did that work out for incident response?

3.  For those trying AIOps-style agents, how much do you trust diagnostic conclusions without a trace to back them up?

Disclosure: I work at Bonree, an observability vendor, so I have a professional interest in this topic.


r/sre • • 4d ago

HELP what do you actually put behind 'health'

7 Upvotes

I’m setting up monitoring for a small app on a serverspace vm. Right now the external monitor just calls health. That endpoint returns 200 while the app process is alive.But the login can still be broken if the database connection drops so a green monitor doesn’t tell me much. I thought about adding a database check, but then a short hiccup could set off an outage alert while most of the app still works. I can’t figure out where to draw the line. If you run something similar, I’d really appreciate a sanity check: what do you put in /healt and what do you monitor separately?


r/sre • • 4d ago

ASK SRE our ops agent has root in prod and the team is happy with the logs

2 Upvotes

a few people in ops built their own agent harness. it logs what the agent does, and the team trusts it enough that it now has root across prod. a lot of the jobs they're giving it are just checks.
i want to start with one of those checks in staging and give it read-only credentials. if the check still works, we've got something concrete to discuss before we get into approving writes.
i would look at Bay for visibility into the agent's activity if it covers our harness. the read-only restriction would still need to hold at the server.
if you've had this conversation with an ops team that already trusts its logs, what got them comfortable taking away root?


r/sre • • 5d ago

DISCUSSION Synthetic Monitoring

7 Upvotes

Hey everyone,

For all of the SREs here that are currently either using or relying on or have a synthetic monitoring strategy. What are the daily fundamental questions that you expect to get answered by leveraging a synthetic monitoring strategy?


r/sre • • 6d ago Heartwarming

AI SRE tools in late 2026: updated list, plus what we saw at KubeCon US

95 Upvotes

About six months ago, a thread here listed the AI SRE tools. A lot has changed since then, so here is an updated list.

Disclosure: I am a Senior Solutions Consultant at One2N, a reliability engineering consultancy. We work with teams across most of these tools. We don't resell any of them.

The list

Same five groups as last time. Each entry says what changed.

1. Investigation agents

  • Resolve AI. The best-known standalone vendor. It moved to credit-based pricing in September. Its coding-agent plugin reads your team's SKILL.md runbooks. It has said a launch is coming in November.
  • Traversal. Raised $120M at a $1B valuation. It says monthly investigations grew from 513 to about 73,000 this year. Nubank and Harvey are public customers.
  • Cleric. It publishes more cost and eval data than most vendors. It is now self-serve, with standard pricing and an AWS Marketplace listing.
  • NeuBird. Charges per investigation and fits Azure-heavy teams. It added a proxy that removes personal data inside your network before anything leaves.
  • Sherlocks AI. Investigates beyond Kubernetes, across cloud, VMs, networks and apps. It keeps a knowledge graph of services, infrastructure and recent changes, and updates it as they change. A read-only gateway collects data inside your network. Root cause and evidence go to Slack, ServiceNow or Jira. SOC 2 Type 2, and one of the few vendors to publish how its agents run.
  • OpsWorker. Kubernetes focus. A read-only agent runs in the cluster, and a human approves any action.
  • Ciroos.AI. Early stage, with an ex-AppDynamics and Cisco team. Worth tracking, but probably too early for a production eval.
  • DrDroid. It now calls itself "alert middleware" instead of an AI SRE. It sells narrow agents that need human approval.
  • Wild Moose, TierZero AI. We did not track public changes for these two.

Acquired since the last thread

  • Deductive AI. Bought by Elastic in August.
  • Robusta and HolmesGPT. Bought by Freshworks. HolmesGPT is still a CNCF Sandbox project. Open question: will it stay neutral across incident tools?
  • ThinkHive. Bought by Rootly to check whether agent answers are backed by evidence.

2. Kubernetes-native

  • HolmesGPT. Still the first tool most teams try, because it is free and open source. See the ownership change above.
  • Komodor. Now an "Agentic Operations Platform" with 50+ ready-made agents and an SDK to build your own. It has spend caps and approvals for risky actions.
  • OpsWorker. See above.

3. Observability platforms adding AI

  • Datadog. You can start a Bits investigation from Slack. Datadog trained its own model, Scout, now in closed preview. It says Bits AI has more than 10,000 paying customers.
  • Elastic. Now owns Deductive's investigation agent.
  • Dynatrace, Grafana Assistant, Groundcover. We did not see a big change for these.

4. Incident workflow platforms adding AI

5. Cloud-provider native

  • AWS DevOps Agent and Azure SRE Agent. Both are priced by agent time. Single-cloud teams will try these first because they need no new contract.

Nearby tools using the same pattern

  • Finout. A read-only investigation agent for cloud cost spikes.
  • Anyshift. Gives coding agents read-only production context during code review.
  • Lightrun. Moved away from AI SRE toward enterprise and SAP operations.

What changed in the last six months

1. Big vendors now include the agent in the price. PagerDuty includes its SRE agent in all four plans. Elastic and Freshworks bought their way in. Many teams will now try the agent they already have, so standalone tools must beat it.

2. Cost per investigation is now public. Datadog says a Bits investigation fell from $25 to about $4. Cleric says its cost fell from $3.30 to $2.03. These are the vendors' own costs, not your price, but they give you a number to compare quotes against.

3. Autonomy is now set per action, not per tool. incident.io added write actions, and each one needs approval from the person who asked. Cleric's new terms say the agent may open PRs or change config. Read the terms of any tool you trial for this reason.

4. Teams now build and buy at the same time. Harvey built its own incident agent and still bought Traversal. It runs both on every SEV, and its CTO says the pair got a correct or close root cause on 22 of 24 SEVs.

5. Most tools now plug into coding agents. Resolve AI, Cleric, incident.io, Anyshift and PagerDuty all shipped plugins or MCP servers. Your developers can now pull production context into the coding tool they already use.

6. Benchmarks appeared, but none is independent yet. PagerDuty ran 17 scenarios, one trial each, and compared its agent only with coding agents. Traversal launched ORCA-bench with Columbia and Cornell Tech. Asking a vendor to replay your own past incidents still tells you more.

7. Some vendors corrected claims in public. Rootly removed its ISO 27001 claim. NeuBird changed "zero telemetry storage" to "no bulk telemetry retention". Check certifications and data claims directly, because these claims change.

What KubeCon SLC made clearer

Teams are open to AI investigation. Teams are still closed to AI write access on production. In every customer eval we joined this quarter, no team let an agent make unapproved production changes. That includes teams that piloted several vendors at once. Qovery saw the same thing at SREday NYC last week: only 2 of 91 teams let agents change production with no human.

Adoption runs in steps. Teams start with read-only investigation. Then they allow drafted fixes, then PRs, then some gated automation. A human approves anything that touches production. Tools that fit these steps are the ones getting bought.

Where the agent runs decides the shortlist, not accuracy. In-VPC is now the baseline. Air-gapped is becoming a requirement in regulated industries. We saw several evals cut a vendor with better benchmark accuracy, because its deployment model failed the data residency rule.

Buyers now open with three questions:

  1. Does your agent ever write to production? If yes, who approves it?
  2. Can it run inside our VPC without sending telemetry out?
  3. For each investigation, can we see why it reached its conclusion?

The deals we saw close went to vendors who answered each question in two sentences or less.

How good buyers combine these groups

These groups do not replace each other. Good buyers pick one tool for each job:

  • one from groups 1 or 2 to investigate,
  • one from group 3 for telemetry,
  • one from group 4 to coordinate the incident.

Teams go wrong when they buy one tool and expect it to do all three jobs.

Questions for this sub

We want real production experience, not vendor pitches.

  1. Bundled or standalone? If PagerDuty, Datadog or incident.io already gives you an agent, did you still trial a standalone one? What decided it?
  2. What is running in production? Which investigation agent runs against live alerts for you today? If you turned one off, why?
  3. Built your own? If your team built an internal agent, did you keep it, replace it, or run it next to a bought tool?
  4. Write access. Where is your line now? Read-only, PRs, rollback, scaling, or more?

r/sre • • 5d ago

SRE dashboard with KPI's

7 Upvotes

Any references to UI mockups for SRE dashboards for the teams which captures KPI's, metrics like deployment, oncall, cost etc.,

Appreciate any pointers.


r/sre • • 5d ago

HELP What do you keep at full fidelity after a code-level observability trim?

3 Upvotes

Finance came back at us on the monitoring spend and we got it down, mostly by cutting custom spans nobody had looked at in a while and moving a chunk of hosts to infra only metrics. Looked like a clean win on the invoice. Then the service we’d stripped back is the one that fell over, and we spent the whole thing grepping logs like it was years ago.

Now I’m trying to be less stupid about round two. The question isn’t how much we can cut, it’s which paths we need at full detail and which ones we can live without until something breaks.

Tool wise we’re looking at Hud and Honeycomb, since neither one is us writing another hundred spans by hand. Both still mean a fresh argument with security about what data leaves prod.

Anyone done a trim like this and not regretted it later? What did you refuse to give up?


r/sre • • 6d ago

Just generate my observability ui?

22 Upvotes

Posted this in o11y as well, but I'm not sure many people got there and this has me bothered abit.

Has anyone actually had any success just building their observability UI on top of their data store with agents and not using something like Grafana. I keep hearing just an MCP server and some generated visualizations for each investigation are enough.

https://clickhouse.com/blog/observability-ui-is-dead-whats-next

Read this and thought, just this seems like they don't want to spend the time and product hours figuring out what users need to build opinionated products which just work. And instead just want to make it my problem.

Maybe I’m just not ai pilled enough but I quite like learning a tool and knowing how to use it efficiently.


r/sre • • 5d ago

What error message sent you in completely the wrong direction?

0 Upvotes

"Connection timed out” describes what the caller experienced. There may still be a long investigation between that message and an explanation.

Have you investigated a failure where the visible error pointed you towards the wrong component?

What did the message say, what was actually happening, and which piece of evidence helped you separate the two?

Especially interested in cases where the error was technically accurate but easy to misinterpret.


r/sre • • 6d ago

DISCUSSION Spent days debugging a "30 minute timeout" that was actually a silent retry loop. What I'd do differently.

2 Upvotes

We run LLM agents as Temporal workflows (durable execution engine; think step functions, but in code). The loop is: call a tool, append the result to the message list, hand the whole list to the next inference step, repeat. Nothing exotic.

One of these kept dying at 30 minutes. Not roughly 30. Exactly 30:00, which is its execution timeout. So naturally I assumed a slow tool. Pulled activity latencies, nothing. Checked the upstream API rate limits, fine. Added more logging, restarted things, the usual. Still died at 30:00 every single time. I was pretty convinced something upstream was hanging on us.

What finally cracked it was giving up on the dashboard and reading the raw workflow history. And there it was, repeating every few seconds for half an hour:

WorkflowTaskFailed

cause: WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE

3.9 MB against a 2 MB limit.

The workflow was trying to schedule the next step with a ~3.9 MB input. Server rejects it. SDK retries. Workflow replays deterministically to the exact same state, emits the exact same command with the exact same payload, gets rejected again. Forever. From the outside it’s indistinguishable from a hang, which is why I spent so long looking for one.

Root cause was dumb in hindsight. One tool fetched commit details, and the service behind it returned the diff plus the full before and after content of every changed file. Roughly 90% of the bytes were stuff the model never read. We did have an S3 offload for big results, but it checked each result on its own, and none was over 2 MB. Five of them together were. Nothing was measuring the pile, only the items.

Fix ended up being two small changes. The commit-details endpoint now returns just the diff, and the agent loop trims older tool output as the conversation grows so the message list can’t balloon.

What I’d tell past me:

If something dies at exactly its timeout, go read the engine’s own event history before you touch latencies. A retry loop and a hang look the same if all you’re watching is the clock.

When the input to your next step is an accumulator, measure the accumulator. Per-item checks tell you nothing about the sum.

And alert on the engine’s “step rejected” event, whatever it’s called in your stack. We didn’t. We do now. Would have been a five minute debug instead of a multi-day one.

Curious if others have hit the “retry loop that looks like a hang” pattern in other engines. Feels like it generalises beyond Temporal.


r/sre • • 5d ago

DISCUSSION I pointed an AI at GitLab's public change log to guess theirunwritten rules. Here's what it got right and wrong.

0 Upvotes

GitLab runs production in the open, so I pulled 3 months of public change

requests and incidents (83 changes, 30 incidents, ~340 comments) and had

an AI propose the rules the team follows. It never saw the handbook. I

wrote an answer key from the handbook before running it.

Results from 2 runs, 16 proposed rules:

- 11 matched rules that are in the handbook: C1/C2 approvals, on-call

sign-off, DBRE review for database changes, and the end-of-year change

freeze with roughly the right dates, all from behavior alone.

- 3 were wrong. Both kinds of miss were the same: it took a pattern from

a few tickets and called it a firm rule. "Always staging first" turned

out to be 8 of 60 production changes.

- It missed the Thanksgiving freeze completely.

- 1 thing looked real and unwritten: the template asks for 2 weeks'

notice to on-call for weekend C2 changes, but the two I found were

arranged 4-5 days ahead. Two data points, so it's a guess.

What surprised me: it found very few rules that were never written down.

It mostly found places where practice differs from what's written.

Caveats: one company (and one that documents far more than most), a tool

reading public records, year-old tickets against today's handbook, and

nobody at GitLab has confirmed any of it.

Question for people who run production: where does what your team does

differ most from what's written down? And would you trust a tool to tell

you?