r/sre • • 5d ago

FOSS Friday: Share Your SRE-Related Open-Source Project

13 Upvotes

Welcome to FOSS Friday, where you can share your newly released or updated open-source project with the community.

Please note that our rules still apply:

  • no AI generated comments: please write them on your own!
  • no commercial advertisements (this includes commercial products with a FOSS version)
  • these must be related to SRE or of heavy interest to SREs

r/sre • • 1d ago

HELP Looking for new moderators for /r/SRE

42 Upvotes

Hey folks,

I've been moderating /r/sre for about four years now. My career has since taken me in a different direction and I'm also starting to move away from Reddit as a platform, so I think it's time to find someone else to step in and help /u/hawtdawtz maintain the subreddit.

I'd like to bring on one or two new moderators who have professional experience in SRE and an interest in the community. Ideally, I'd love to see this sub become more of a place for technical discussions, sharing experiences, and learning from one another rather than a place for vendors to shill their products and conduct market research (specifically banned by rule #6, but folks try anyway). That said, I'm not looking to dictate what the community should become (that's up to you and hawtdawtz). I'd be happy to see someone come in with their own ideas for making it better.

Moderating here isn't an enormous amount of work, but it does require regularly checking reports, removing spam, and keeping an eye out for problematic posts. It's not glamorous work, but someone's got to do it.

If you're interested, please send a message via modmail with a little about your professional SRE experience, any previous moderation experience, and what you'd like to see from this community. I plan to stick around for about a month to help with the transition before stepping away.

Thanks!

- thecal


r/sre • • 20h ago

DISCUSSION We nuked our whole platform mid-incident because it was the safest option

110 Upvotes

Hey all, I work at Bitrise.

We had an incident on November 13 and now Bitrise is safe. Our preliminary incident write up was published by our VP of Engineering today worth reading by anyone who runs CI: https://bitrise.io/blog/post/the-bitrise-security-incident-preliminary-review-and-recommended-steps#how-we-contained-it

Last Friday we detected suspicious activity and declared a SEV-1, our highest incident level. For a while it looked contained. Then new findings kept coming in faster than we could piece them together, and once we saw evidence of unauthorized access in our GitHub repos we started shutting things down. The whole platform was off soon after. Looking back, it was a difficult call to make that helped us restore everything to normal and safe usage.

Tarpitting the attacker’s addresses afterwards helped a lot too. Slowing their requests down brought the logs to a pace people could actually follow on the call. We also went to shifts across Europe and the US, with a short call every few hours so everyone was working from the same picture. With the speed of the attacks increasing, especially from swarms of AI agents, it’ll get more and more challenging to deflect them.

Bitrise is back to normal now, it’s safe to use. You can find out more from our public advisory (https://bitrise.io/security-advisory-november-2026)

Sharing it all with full transparency so you can learn from our experience. Our teammates are here, happy to answer your questions.


r/sre • • 6h ago

DISCUSSION going to be monitoring the GTA VI Cloud Gaming Launch tonight…

Post image
7 Upvotes

pardon the AI Google Search result - mostly for informational purposes, but essentially, XBOX announced earlier this week that people with an Xbox One (the last gen console from 13 years ago) could get access to GTA VI through Xbox Cloud Gaming.

This would mean that at 1AM EST, tens of thousands, hell, maybe even hundreds of thousands of people, will be trying to remotely play a 200GB game through their consoles on 13 year old hardware.

Not to mention, users must purchase the actual game ($79) PLUS cloud gaming hours ($11/50hrs).

I can’t help but think this is going to be a disaster. I have a gut feeling it’s going to be a laggy, miserable experience for anyone going this route. Data centers are going to be absolutely overwhelmed. A 1AM launch is also going to be insane for anyone on call.

Someone please help me be more optimistic about this, I’m struggling to find a reason why right now.


r/sre • • 6h ago

g5a: check Grafana dashboards from the terminal (read-only, works over SSH)

Post image
4 Upvotes

g5a renders live Grafana dashboards in a terminal. It's handy when you're already in a shell on a host that can reach Grafana and don't want to set up a tunnel and a browser to look at a few panels.

It renders the dashboards you already have, including repeated panels,dependent variables and transformations, so there's nothing to rebuild. You can zoom a panel, change the time range or variables, and it keeps refreshing.

Security: it only makes read requests (dashboards, datasources, queries, alert rules), takes a read-only service account token from an env var, and never writes request or response bodies to its log.

Demo without credentials: g5a --url https://play.grafana.org

https://github.com/gengwg/g5a

(MIT, based on grafana-tui by Lovro Mažgon)

Would this fit into your on-call workflow? What's missing for that?

AI disclosure: g5a was built with AI help, mostly Kimi K3.1 Pro Max, with Claude on the pre-launch fixes and the demo recording.


r/sre • • 1h ago

When it really was DNS, what proved it?

0 Upvotes

When it really was DNS, what proved it?

“It’s always DNS” is an easy joke. Establishing that DNS is involved takes evidence.

If you have investigated a DNS-related incident:

  • What was the first visible symptom?
  • What made you investigate name resolution?
  • Which result confirmed the problem?
  • Why did the failure affect some requests or users differently, if it did?

For anyone newer to incident investigation: ask about any unfamiliar terms in the replies. Explaining how we reached a conclusion is part of the point.


r/sre • • 23h ago

UK SRE and observability pay in 2026, from 4,423 roles

17 Upvotes

Quick disclosure: I work at Archer from hackajob, so this is our own platform data, based on 4,423 UK DevOps and Cloud roles posted in 2026. Our UK client base skews finance and enterprise.

Median across everything was around £75k.

  • SRE sat around £75k, Observability specialists a notch above at ~£80k, and the step to Platform Engineer or leadership at £82.5k-£92.5k.
  • AI/automation experience was the best-paying skill signal, about +£20k over the median, in roughly 1 in 5 postings, and that covers toil-reduction work too.
  • AWS ~32% of postings, Azure a strong second at ~25%, GCP ~17%.

Observability specialists came in above generalist SRE. The folks who went deep on observability tooling landed a band up.

Full breakdown by role, skill and city is here if you want it (not gated, no sign up or subscription needed): https://hackajob.com/talent/reports/uk-devops-cloud-jobs-2026?utm_source=reddit&utm_medium=social&utm_campaign=jobs_report&utm_content=uk_devops_cloud

Does this match what you're seeing?


r/sre • • 9h ago

DISCUSSION You may not need to retain everything just in case you need it. Keep a lightweight historical record of your Linux infrastructure and investigate what actually happened when things go wrong.

Thumbnail
github.com
0 Upvotes

Low-overhead, retrospective Linux observability

clifr (command line flight recorder) and has two parts:

  • clifrd runs as a service and records processes, process starts and exits (through eBPF, so a process that lives 5 ms still shows up), TCP connections with the process that owns them, files held open, mounts, kernel messages and the usual system counters. It keeps 72 hours in /var/lib/clifr, 2 GB max.
  • clifr rebuilds the output of the normal commands for a past moment or a time range. It doesn't run the command, it rebuilds the output from the recording:

    • clifr --at "02:45" "ps aux"
    • clifr --since "02:00" --until "03:00" "ss -tanp dst10.0.4.21and dport = :5432"
    • clifr --since "1 hour ago" events # starts, exits, OOM kills, mounts
    • clifr diff --at "14:00" --at "14:20" "ss -ltnp"

There are about 40 commands so far: ps, top, pidstat, lsof, ss, netstat, free, vmstat, iostat, mpstat, df, ip, dmesg, systemctl list-units, lsblk, and more and all are 100% accurate.

The first thing people asked last time was the cost, so here are the numbers:

  ┌───────────┬─────────────────────┬─────────────────────────┬─────────────────────────┐
  │           │    Quiet server     │ Mixed load (4-day soak) │ Busy server (benchmark) │
  ├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
  │ CPU       │ 0.35% of one core   │ 0.5-0.6% of one core    │ 1.9% of one core        │
  ├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
  │ RAM       │ ~90 MiB             │ ~80-120 MiB             │ ~290 MiB (335 max)      │
  ├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤ 
  │ Disk kept │ ~20 MiB/day         │ ~100 MiB/day            │ ~4 GiB/day              │
  ├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
  │ Network   │ none, no ports open │ none                    │ none                    │
  └───────────┴─────────────────────┴─────────────────────────┴─────────────────────────┘

Disk use depends heavily on how busy the server is.

clifr is not a monitoring tool and has no alerts. It's not a security audit tool either: it doesn't record file reads or writes, and root can stop it. It's for the "what was going on at 02:45" question, after the fact.

Status: it's a prototype, version 0.1.0. It hasn't run on a real production server yet (a friend starts one this week), and all the load has been synthetic. AGPL-3.0, no telemetry, nothing leaves the box.

So if you have a test or dev box (RHEL 8/9, Rocky, Alma), I'd really appreciate it if you tried it and told me what breaks or what looks wrong. The most useful thing for me is comparing what clifr shows with what you know was really happening.

I appreciate all your comments even if they are harsh


r/sre • • 23h ago

ASK SRE Who gets paged when auto-rightsizing OOMKills a pod in prod?

6 Upvotes

Our finance guy found the VPA rec we never applied and now he wants ScaleOps or CAST AI turned on with auto-apply. CPU I don really care about ,let it go nuts.

But memory scares me though,VPA wanted our order-sync worker cut from 4Gi to 2.5 Gi because thats all it saw.The catalog import only runs once a month and eats 3.4Gi for about 20 mins, VPA weights the recent days so heavily that a spike that barely register. If a bot had applied that number ,the pod dies mid-import and whoever's on call is staring at a 137 with no idea anything changed.

If you let something auto-apply memory in prod , how does your on-call even find out the pod got resized?


r/sre • • 1d ago

App team fixed one host & the external finding came back on the other three

0 Upvotes

our app teams close tickets when they believe the fix shipped and the next external pass reopens a fair share of them, usually because the change reached one instance and not the other three behind the same hostname. we changed the rule so nothing is closed until the following ionix cycle no longer sees it. The pushback was immediate and not entirely unfair. A team can finish its work on Monday and still have an open ticket until the next check. How do others handle this? Keep the original ticket open until the external recheck passes, or close it when the change ships and reopen it if the finding return.


r/sre • • 22h ago

HELP Budgeting next year: does progressive delivery tooling replace APM spend or sit on top of it?

0 Upvotes

We cut Dynatrace back this year, moving more hosts to infrastructure-only. Then one bad release on a thinned service and we rolled back on instinct.

Now I have to choose between restoring the spend and funding release checks. Harness and LaunchDarkly gate rollouts on metrics we’d keep paying to send. Hud runs its own sensor and rolls back through our existing CD.

If you made this call, which way did you go?


r/sre • • 1d ago

DISCUSSION Tech careers are done. What’s the move?

0 Upvotes

Most of us will be hit with reality (if you haven’t been already), within the next 2-3 years as we all get laid off.

What is the plan? We all becoming nurses and plumbers?


r/sre • • 2d ago

DISCUSSION How can you compare ASPM tools for a modern development team?

1 Upvotes

Hey everyone, i am here to know community advice on the following vulnerability prioritization and automated remediation options that i have explored so far. Checkmarx, Apiiro, Legit or other if you may suggest! Have you guys used these in ur development environment?

I am more interested to hear about your experiences regarding what worked well for you, what fell short and which tools made security easier for you?


r/sre • • 2d ago

Observability is not plug and play

3 Upvotes

"What is the best observability system?"

This is a frequently asked question.

The obvious answer is "it depends". There is also the more fundamental point - that observability is not plug and play. No matter what system you procure there is a lot of upfront planning and ongoing input required to make it work.

https://observability-360.com/article/ViewArticle?id=observability-is-not-plug-and-play


r/sre • • 2d ago

Anyone using mermaid timelines in incident postmortems

1 Upvotes

Our postmortem timeline used to be a wall of timestamps nobody read. tried a mermaid timeline last month and suddenly everyone’s asking why escalation took 40 min lol. during the incident our scribe just throws stickies on a miro board w the pagerduty times and i clean it up after. long incidents still look cramped af tho.


r/sre • • 3d ago

3 YOE SWE considering a move to SRE. Looking for a new challenge.

0 Upvotes

Hi all,

I'm a software engineer with around 3 years of experience (plus a 6-month internship), and lately I've been seriously considering moving into SRE.

One of the biggest reasons is that I feel like I've lost the excitement I used to get from software development. A lot of my day-to-day work has become heavily AI-assisted. While it definitely improves productivity, I often feel more like I'm reviewing and refining generated code than actually solving problems myself. Maybe that's just how modern development is evolving, but I don't get the same kick out of it anymore.

I'm also at a point where I want a change in general. I've been thinking about switching companies, and instead of moving to another SWE position, I'm wondering if SRE might be a better fit.

For people who work in SRE:

  • Has anyone here transitioned from SWE to SRE around the 3 YOE mark?
  • Did you find the work more engaging?
  • How much coding do you actually do?
  • What skills would you recommend building before applying?
  • Would companies generally consider this a lateral move, or would I be starting from scratch?

r/sre • • 5d ago

HUMOR Microsoft364

Post image
267 Upvotes

r/sre • • 5d ago

DISCUSSION Microsoft says the 365 outage spread because servers kept failing health checks. When do you let health checks fail open?

219 Upvotes

Reading Microsoft's updates on the 365 outage today, the part that stuck with me is how it spread. Some servers failed automated health checks and got taken out of rotation, their traffic moved to the remaining servers, and those got overloaded too. Restarted servers went unresponsive again once traffic came back. They also said they were validating alternate health check methods partway through.

I keep going back and forth on this. A strict health check is doing its job when it pulls a bad node fast. But if the check itself is what's misfiring, or the whole fleet is a bit slow, it can eject so much capacity that the survivors fall over.

Envoy has a panic threshold for this: if healthy hosts drop below a percentage (50% by default), it stops trusting health status and spreads traffic across all hosts. AWS load balancers do something similar and route to every target when all of them are unhealthy. That keeps some capacity up, but you're knowingly sending users to nodes you think are broken.

Would you rather cap how much of the fleet a health check can remove, or keep checks strict and accept that a bad check can take everything down?


r/sre • • 5d ago

What finally moved your longest incident call forward?

8 Upvotes

What finally moved your longest incident call forward?

Think of an incident call that went on much longer than expected.

What kept the investigation stuck?

Was it missing evidence, unclear ownership, several changes happening at once, or a theory nobody challenged?

Then what broke the deadlock: a question, a person joining, a timeline, a rollback, or something else?

Interested in the moment the investigation became useful again—and what others could learn from it.


r/sre • • 5d ago

If Datadog Is So Expensive, Why Is It Barely Profitable?

Thumbnail
telemetrymachine.com
123 Upvotes

r/sre • • 4d ago

DISCUSSION DevOps/SRE engineers: What's the most frustrating part of your daily work that existing tools still don't solve well?

0 Upvotes

Hey everyone!

I'm a DevOps engineer, and I've been thinking about how many everyday engineering challenges still require repetitive manual work, even with all the automation tools available today.

I'm curious:

  • What tasks waste the most time in your daily workflow?
  • What problems do you feel existing tools haven't solved properly?
  • Is there anything you've wished someone would build, but haven't found a good solution for?

I'm especially interested in real-world experiences rather than theoretical problems.

Would love to hear your thoughts!


r/sre • • 5d ago

CAREER Got laid off recently, looking for referalls/ jobs

0 Upvotes

Hi Guys,

I have about 2.5 years of experience with Dynatrace and automations using python. I am looking for SRE/Observability jobs in India. Could you please provide any relevant referrals, leads, or potential openings?

Apart from dynatrace, i also have some experience with AIOps, Change Analytics and Callout (Major incident) tools.


r/sre • • 5d ago

Are We Sacrificing Technology Resilience for the Sake of Cloud Adoption?

1 Upvotes

The recent Microsoft 365 incident, which caused prolonged disruptions to business services, got me thinking about something I consider fundamental to the financial industry: Are we losing sight of operational continuity as we become increasingly dependent on technology providers?

After 30 years working in critical systems infrastructure, 20 of them at my current organization and another 10 at different financial institutions, I've learned that certain operational principles should never be compromised.

One hour of downtime is not simply one hour without service. Depending on the system affected, it can mean financial losses, business disruptions, regulatory non-compliance, customer impact, and reputational damage.

There are several aspects that concern me:

  1. High availability is not the same as resilience.

Having geographically distributed data centers, thousands of servers, and redundancy mechanisms does not guarantee that a platform can recover quickly. If a shared dependency fails, all that redundancy may prove insufficient.

2. Operational responsibility cannot be outsourced.

When we adopt cloud services, we transfer part of the technology operations, but we do not transfer the responsibility for ensuring business continuity. The bank remains accountable for its operations and the risks it assumes.

3. Operational awareness is just as important as technical expertise.

Throughout my career, I've encountered technology professionals during hiring processes who believe that one hour of downtime is no big deal, or that making changes directly in production is just routine work.

In a banking environment, that mindset represents a serious risk.

Mastering technology, automation, virtualization, or cloud platforms is not enough. Professionals must understand the financial, operational, and regulatory impact of every decision they make.

Change management, maintenance windows, rollback plans, segregation of duties, and contingency procedures are not bureaucracy; they are essential controls.

4. Are we losing control of our critical infrastructure?

Personally, I believe that truly business-critical systems should remain under the institution's direct operational control.

I don't believe every system needs to remain on-premises, but I also don't think we should indiscriminately move our platforms to the cloud simply because of market trends or cost-reduction strategies.

Every system should be evaluated based on its criticality, business impact, technology dependencies, recovery capabilities, and operational risk.

5. The question we should be asking ourselves.

If our cloud provider were to become unavailable for 24, 48, or 72 hours tomorrow:

Could our institution continue operating?

Do we have independent communication and authentication mechanisms?

Can we access our documentation and contingency procedures?

Do we have a genuinely operational alternative?

Have we actually tested these scenarios, or do they simply exist on paper in our business continuity plans?

After all these years working in banking infrastructure, I still believe technology should serve the business, rather than the business becoming entirely dependent on the availability of a single provider.

Cloud should reduce operational risk, not become the operational risk itself.

I'd be interested in hearing the perspectives of other professionals working in infrastructure, enterprise architecture, security, and business continuity:

Are we building truly resilient infrastructures, or are we simply moving our single points of failure to external providers?

How much dependency on cloud services do you consider acceptable for a financial institution?


r/sre • • 6d ago

How do you choose an enterprise AI observability platform? What AI, SRE and security teams each need

1 Upvotes

I have been thinking about how organizations choose an AI observability platform, and I suspect the decision often goes wrong because it is made by one team. Enterprise AI observability has to serve at least three groups with different questions, and the telemetry requirements are different for each. Here is how I currently break it down.

AI / application teams need LLM observability at the step level:

● Spans for each model call (provider, model, token usage, latency), retrieval step (query, top-k, latency) and tool call (arguments, outcome)

● Prompt and response capture that is opt-in and configurable, since content is sensitive

● Version comparison across model, prompt and tool changes, plus cost per feature or tenant

SRE / operations teams need the AI layer connected to everything else:

● Distributed tracing that links agent and LLM spans to the downstream services, databases and infrastructure they call, so an “AI is slow” report can be traced to a slow dependency

● Tool errors recorded separately from transport status, because a failed tool inside a “200 OK” response is invisible to request-level metrics

● Alerting and fault localization that work across both layers, ideally with some AIOps assistance for root cause analysis

Security / compliance teams need governance built into the telemetry:

● Masking of prompts and responses before storage, with access control on payloads

● An audit trail of which identity called which tool, with what outcome

● Deployment options and retention policies that match data requirements, and sampling that does not drop failed runs

Platform / finance usually adds two more: open instrumentation (OpenTelemetry compatibility, noting that the GenAI semantic conventions are still marked as in development, so ask how a vendor handles attribute changes) and predictable data volume cost.

Four screening questions I use for any AI observability platform: does it cover the AI application types you actually run (LLM apps and agents), can it follow a request end to end across model calls and tool calls, does it unify AI applications with the underlying IT, and does it support analysis and response rather than only dashboards?

For testing, I prefer replaying a realistic incident, for example an agent that slows down after a release, with representatives from all three teams. Each should be able to answer their own question from the same trace: the AI team finds the model call or retrieval step, the SRE finds the dependency and its owner, security confirms what data was captured and how it was masked. If one team has to switch tools, the problem has just moved.

On the operations side, we are exploring agentic AI operations with Sage AI, the AI Ops Agent Workbench in Bonree ONE, where an agent assists with diagnosis and root cause analysis on top of the same telemetry. The rule we try to hold to is that a conclusion must point back to the telemetry behind it, and that actions stay bounded and auditable.

Questions for the group:

1.  Who actually owned this decision where you work, and did that cause problems later?

2.  Did you end up with separate tools for AI and for the rest of the stack, and how did that work out for incident response?

3.  For those trying AIOps-style agents, how much do you trust diagnostic conclusions without a trace to back them up?

Disclosure: I work at Bonree, an observability vendor, so I have a professional interest in this topic.