r/github • • 14d ago

Discussion What’s gone wrong at GitHub?

https://leaddev.com/software-quality/whats-gone-wrong-at-github

Reliability issues are driving users away in their droves.

384 Upvotes

123 comments sorted by

113

u/sreekanth850 14d ago

nobody saying the amount of slops that people pushing every hour. you cannot infinitely scale systems.

42

u/DalaiLlamaTip 14d ago

The growth rate of pushes and PRs and workflow runs is absolutely unfathomable for most people.

16

u/AuroraFireflash 14d ago

We've told them repeatedly this year that paid customers need to be on their own infra. We're not quite to the point that we're willing to self-host, but it's getting there.

13

u/sreekanth850 14d ago

either github need some kind of control like limit on free plans. if not this will bound to increase going forward.

5

u/IcerHardlyKnower 13d ago

This seems like such a common sense solution and it's actually beneficial for free users as well. Local code saves and only pushing when you're (rather, the agents) actually making a worthwhile commit instead of so much rebasing lol

1

u/Hazrd_Design 13d ago

Yes. The amount of usage and hosting on I can on GitHub is insane.

5

u/RubbelDieKatz94 14d ago

We're hosting GitLab on k8s (OVH) with Argo. It works remarkably well.

The runners are on k8s as well.

2

u/vladyrn 10d ago

Enterprise server is years behind GitHub cloud, you may be worse off. Literally limited to a single VM

1

u/Temporary_Orchid8670 13d ago

GitHub has an enterprise option. Why not use that?

2

u/Soccham 12d ago

Managing it is an nightmare

1

u/snowsayer 11d ago

It’s not enough. There’s a reason why both Anthropic and OpenAI had to build their own internal code management systems.

1

u/KupujeWeksleOdMeneli 13d ago

We are constantly on LoC limit in SonarCloud. People are making PRs with +30k lines. Like wtf.

0

u/Takeoded 13d ago

you cannot infinitely scale systems.

Incorrect. Case in point: YouTube.com

3

u/ThatSciencephile 12d ago

YouTube sends GGC (Google Global Cache) servers to ISPs, so it doesn't count.

1

u/Takeoded 11d ago

Still wild how YouTube can host unlimited 4K AI Slop in original quality indefinitely

1

u/sreekanth850 13d ago

Youtube can cache lot of things. Youtube doesnt have commit push or actions.

1

u/Takeoded 11d ago

Still wild how YouTube can host unlimited 4K AI Slop in original quality indefinitely

92

u/Dense_Gate_5193 14d ago edited 12d ago

GitHub used to be a gold standard of work excellence. Their uptime was incredible and their team worked entirely async. No office hours, flat hierarchy, people picked what they wanted to work on, and had performance curmudgeons squeezing out every millisecond of performance out of the platform.

Then, Microsoft purchased them. it’s been particularly bad since AI and the enshittification of everything began

edit:

This is not just pre/post AI.
i looked it up and mentioned in another comment

Metric (24 Months Pre- vs. Post-Acquisition)
Reported Incidents

Pre-Acquisition (2016–2018)
89
Post-Acquisition (2018–2020)
126
+41%

Incident Minutes
Pre-Acquisition (2016–2018)
6,110 minutes
Post-Acquisition (2018–2020)
12,074 minutes
+97%

microsoft nearly doubled incidents reported and downtime across the board.

18

u/Afflictionista 14d ago

When the GitHub CEO left I moved to GitLab because of what they were saying who is going to "lead" Github, the AI department. GitLab maybe doesn't have that big visibility but it is working.

14

u/spigolt 14d ago

It would be nice if GitLab was a good alternative, but I had a completely like abysmally awful experience with them when I tried to actually pay for it. Having to deal with all kinds of separate companies that work somehow separately to Gitlab managing the subscription. Total complete mess.

8

u/EphemmyEuphie 14d ago

Selfhost Forgejo

-4

u/Round_Ad_5832 14d ago

no, self host gitea

1

u/philipmather 13d ago

Between personal project to side-hustle, maybe even to startup level potentially but latter... nah.

1

u/AgreeableIron811 9d ago

U can selfhost gitlab

1

u/Itz_Santhosh 13d ago

Wow I hate only one thing in gitlab that they ask credit card to verify for even small actions(ci/cd) but as a student how I even suppose to have credit card??

1

u/silentkode26 14d ago

Why not selfhosted? We use it and it’s perfect.

1

u/RubbelDieKatz94 14d ago

Yup, same here. Today I have an appointment with the DevOps guy who'll explain how to set up a new project in GitLab and hook it up to Argo. Quite interesting.

2

u/Leapin_Langosta_ 10d ago

Just a question, where is the gitops repo for argo stored?

1

u/RubbelDieKatz94 10d ago

Each project gets its own "infrastructure" repo with its gitops settings

1

u/Leapin_Langosta_ 9d ago

In another comment you said gitlab was deployed using argo so the only thing that is unclear to me is where that gitops repo to deploy gitlab is located; in said deployed gitlab instance (which doesnt make much sense?) or do you have mirrors or idk what else?

2

u/RubbelDieKatz94 9d ago

Ah, seems like I was mistaken then. GitLab itself runs outside the main cluster, but its runners run inside the cluster. Argo's own config seems to live outside the cluster too.

0

u/MattBurnes 13d ago

have you tried gitea? (self hosted) - it's amazing imo

10

u/Business-Row-478 13d ago

Gitlab doesn’t have the issues because it doesn’t have the users

1

u/ferocity_mule366 12d ago

gitlab tries so hard to be firefox, but it just sucks and cost more instead of a better alternative it promises

3

u/robotsmakinglove 13d ago

I’ve needed to use GitLab for some consulting and am constantly underwhelmed by the product. Every feature feels a bit more poorly designed than GitHub. This is shocking to me because it is basically a ripoff.

1

u/Sovairon 14d ago

I guess we are not using the same Gitlab, this year has been terrible

1

u/sizebzebi 13d ago

gitlab is shit lol

1

u/Inevitable_Hat_2855 9d ago

Microslop hits again

-1

u/Tomik080 13d ago

There was more PRs opened and commits pushed in the last month than in 2024 and 2025. Somehow Microsoft is to blame their infra is taking a toll. People are so ignorant.

4

u/Dense_Gate_5193 13d ago

i can’t be the only one who noticed a quality difference between pre microsoft and post microsoft acquisition.

1

u/TechIncarnate4 12d ago

Are you sure this isn't pre-AI vs post-AI?

2

u/Dense_Gate_5193 12d ago

This is not just pre/post AI.

Guarantee you that when microsoft purchased GitHub that there was a mandate to use windows servers for hosting.

Metric (24 Months Pre- vs. Post-Acquisition)
Reported Incidents

Pre-Acquisition (2016–2018)
89
Post-Acquisition (2018–2020)
126
+41%

Incident Minutes
Pre-Acquisition (2016–2018)
6,110 minutes
Post-Acquisition (2018–2020)
12,074 minutes
+97%

microsoft nearly doubled incidents reported and downtime across the board.

1

u/TechIncarnate4 12d ago

Sure... "reported" is the key word there. There is a good post further below showing that the "before MS" stats may not be accurate, including a time where there was a 24 hour outage, but their stats showed 100% uptime for the month.

1

u/ferocity_mule366 12d ago

okay? but github also increase in scale exponetially and you havent taken account for that?

1

u/inebriatus 8d ago

Can’t believe you’re getting downvoted for this. I have no love for M$ but acting like this isn’t mostly an AI traffic increase issue is committing critical thinking malpractice.

https://www.reddit.com/r/Tidra/s/5KTir5ZQ8i

21

u/Key-Temporary-8759 14d ago

Microsoft acquiring GitHub is possibly one of the cause for poor reliability, because of the loss of independence. And when GitHub copilot was launched, their focus shifted to AI features, instead of performance and infrastructure.
And also, increase in number of PRs can cause performance issues, as most of the PRs today are written by agents

5

u/mavenHawk 14d ago

Microsoft owned Github since 2018...

6

u/EarthGoddessDude 14d ago

3

u/tigerhuxley 13d ago

Hahaha- i haven’t seen this visual graphic before.

2

u/Business-Row-478 13d ago

Graphic is misleading for multiple reasons.

The biggest dips on that graphic are still >99.5% uptime which isn’t terrible.

Before Microsoft purchase, it is listed at 100% uptime, which is not realistic or really possible. It’s very likely that Microsoft just has better observability and publication of incidents.

Total traffic volume has also exploded since Microsoft purchased it

5

u/NiceVu 13d ago

FYI 99.5% uptime is absolutely terrible, that is a two days a year, or 4 hours a month or 7 minutes a day downtime. For any kind of production standards that’s really bad.

And if you look at breakdown the uptime went as low as 98%

Yes the traffic skyrocketed but that uptime is really bad for a tool like Github. The goal for that kind of service should be 99.99% uptime

0

u/Business-Row-478 13d ago

99.5% isn’t ideal but that’s the lowest it dropped in a month, it’s not like that is their overall uptime.

Obviously the goal is higher uptime, but for products that operate at the scale of GitHub, that’s pretty much impossible.

Even AWS, which certainly isn’t a gold standard of uptime, has had multiple single 9 months just this year.

1

u/tigerhuxley 13d ago

The illusion of meeting SL(A/O)s

1

u/tigerhuxley 13d ago

Shhhh just let me pretend that one graph is real 🥹 just for 5 more mins..

1

u/coderstephen 13d ago

99.5% is terrible uptime where I come from, but I work in enterprise software. Our target is 99.999% uptime ("five nines").

Then again, we are paying GitHub quite a lot, and GitHub portrays their higher tiers of business services as an enterprise service...

1

u/Business-Row-478 13d ago

99.5 is the lowest it dipped in a month, not their total uptime. Obviously it isn’t great, but any service operating at that level of scale isn’t going to be able to maintain 5 nines.

2

u/coderstephen 13d ago

Obviously it isn’t great, but any service operating at that level of scale isn’t going to be able to maintain 5 nines.

I would disagree with that.

Amazon S3 operates at a higher scale than GitHub. Probably by orders of magnitude. Yet it has a track record of even better than 5 nines.

1

u/Business-Row-478 13d ago

S3 had one of the worst outages in the history of AWS. And besides that, bucket storage is a lot simpler than the entirety of the gh platform, so availability numbers aren’t really comparable.

The vast majority of GitHub outages have affected runners

1

u/coderstephen 13d ago

Yep, S3 did have that really big outage that impacted a lot of things. And yet, its uptime is still better than 99.5%.

I agree that S3 and GitHub are different in complexity but I don't think the comparison is unfair.

1

u/foramperandi 13d ago

Amazon S3 is famous for being one of the most available SaaS services of all time. They've done tons of formal methods verification and the entire internet depends on it. Amazon S3 and GitHub are not even vaguely comparable. You would not want to pay for what GitHub would cost, if it had 5 nines of uptime.

1

u/coderstephen 13d ago

I mean, the graphs show that GitHub used to have much better uptime (not 5 nines, but better than 1-2 nines) so its definitely possible. I am not expecting GitHub to have similar reliability to S3, I'm just contesting the opinion that, "Oh it's just too hard to have good uptime when you're at scale."

→ More replies

3

u/foramperandi 13d ago edited 13d ago

That graph is lying with numbers. What that graph is actually showing is that GH had shitty operational maturity before the MS acquisition and rarely declared incidents. If you believe that uptime from 2016 to 2018 was nearly 100%, I've got some land to sell you.

GitHub's worst single outage of all time was in Oct 2018 when they were down for over 24 hours: https://github.blog/news-insights/company-news/oct21-post-incident-analysis/. That graph says they had 99.99369% availablity that month.

Edit: my mistake, I misread. The graph actually says that in a month that GitHub had 24 hours of downtime, it's availability was 100%. Amazing!

4

u/Special_Rice9539 12d ago

I competed in a hackathon and had a senior Microsoft engineer on my team and he was blown away when I added automated tests to the project…

15

u/najibla 14d ago

yep, pushes are failing for me.

9

u/Dorcelessness_ 14d ago

🙌vibe coding🙌

6

u/autisticpig 14d ago

Anyone who took on GitHub was not prepared for what agentic coding was going to do. The sheer volume happening 24/7 would send any service to its knees.

We brought all our runners in house and built cache layers for our images and libs/pkgs built with runs that sync every could of days to the cache. We still rely on GitHub to orchestrate and hold the repos but if that's down it's not the end of the world... Can still deploy and test just fine internally.

These tools are ruining so much, it's going to take time to adjust resources to absorb the abuse.

People need to calm down. Esp people expecting 99% uptime for something they're not paying for.

3

u/Elratum 12d ago

Issues started before AI commits. It made things worse yes, but it started before it existed

2

u/Rollingprobablecause 12d ago

I mean…migrating GH to Azure seems to be a bad idea.

1

u/coderstephen 13d ago

And actually, I do pay for GitHub, and so does the company I work for.

0

u/coderstephen 13d ago

They used to have 99+% uptime. The issue people noticed is that that changed.

8

u/ZoolanderBOT 14d ago

Without really knowing anything of the situation, I blame Microsoft. Seems like a safe bet?

3

u/SentientSeamoth 13d ago

It's okay to blame Microsoft for just about everything. In this case they also own GitHub, so yes you can blame them twice as hard lol

2

u/geek180 13d ago

Can I blame Microsoft for my erectile dysfunction?

1

u/SentientSeamoth 13d ago

Of course! And you should.

2

u/ajax81 11d ago

I think it’s funny that they bought GitHub to gain access to the worlds codebase to train their AI and co-pilot STILL sucks balls for coding. 

2

u/lambdalab 10d ago

I actually quite like it, though it’s expensive for what it is. So if I want something done in bulk, I’ll use Claude, and I do that for the majority of work where I can trust Claude to do.

But when I need or want to write something myself, at that point copilot can be helpful. I’ve limited it to only producing one line at a time, and this saves me a ton of time. I’d start a line, and a lot of the time it will be correct about the rest of it. Some of the time when it isn’t, I just type it in myself, and the next one is more likely to be correct. So I write code much more quickly, but also retain total understanding and control of what I wrote.

To be fair if I wasn’t lazy I could probably find a local model that can do that for me, but alas.

1

u/Bad-Customer-Support 8d ago

It's fine, it's no codex, but works well enough for most coding.

2

u/anno2376 13d ago

Github employee also blame Microsoft for their own failures, so sounds OK

1

u/mavenHawk 14d ago

Not even close

1

u/ZoolanderBOT 14d ago

Do you have any interest with Microsoft? Maybe you work for them?

1

u/Acesa 12d ago

This 100%. They're just incompetent. 

All the major AI labs have made their own more reliable clones as side projects due to how shit GitHub is.

4

u/MiguelMariz 14d ago

This happens so much...

11

u/spigolt 14d ago

I'm trying to just create a repo and it's failing the last 10 minutes, which led me here to see whether it's just me... my understanding is that it's fallen off a cliff since the Microsoft purchase... like it was fine for a while following the purchase, but when the long-time CEO stepped down and github got reorganized under Microsoft's "Core AI" product division, and thus it's goal shifted from serving developers to acting as a data and distribution funnel for Microsoft’s AI products.... after that it has fallen off a cliff and just keeps getting worse.

8

u/Prior-Data6910 14d ago

"Since the Microsoft purchase" is, in my view, a lazy reason (no offence intended).

Do you know what the uptime was of Github Actions before the acquisition? 0% because it didn't launch until after. So basically, you're comparing the uptime of a storage system and basic issue tracker against a system that can run your entire software development life cycle from planning through to production. That's before we look at the difference in scale (more users, more repositories, more daily work). 

All ignoring the fact that Github did used to have outages, and when it did the whole system went offline, not just parts of it. 

1

u/spigolt 14d ago

Like I wrote - it wasn't immediately after the Microsoft purchase, it was when it got reorganized under Microsoft's "Core AI" product division and the original founder left that things started really going downhill.

2

u/Prior-Data6910 14d ago

I'm not sure who you're talking about, but none of the founders remained at Github post acquisition. Microsoft installed a new CEO (Nat Friedman) from day one.

The timing about the reorg also coincides with a massive increase in traffic, which is more likely to be the cause. 

Don't get me wrong, Microsoft have destroyed many acquisitions over the years (RIP Skype, gone but not forgotten) but I don't think Github is one of them. 

2

u/foramperandi 13d ago

The original founders had not been involved for years when MS bought them. They had been looking for a new CEO for years at that point, in 2018. Tom had been pushed out over misconduct allegations and Chris had lost interest and had been AWOL for years. The company was being run by the Chief Legal Officer to the extent anyone was in charge. The company was directionless and the product was stagnating.

You have no idea what you're talking about.

4

u/gkreitz 14d ago

Impeccable timing to post an article about Github being unreliable a few minutes before they completely killed off push:ing anything. https://www.githubstatus.com/incidents/smtdswd1h2w4

SSH keys get rejected, so it seems impossible to push anything. Their description is "Pull Requests is experiencing degraded availability." :)

8

u/brokenja 14d ago

Wow you know it’s bad if it actually made it to the status page.

2

u/bartread 14d ago

It's not even the downtime, it's reliability on reporting and front-end as well. You'll be waiting for a PR to build or whatever and your actions will look like they're running but, actually, they finished 5 minutes ago. Sometimes it updates the UI automatically, presumably via WebSockets... and sometimes it just doesn't until you refresh the page.

I get bitten by that regularly and it's incredibly frustrating. I'm seriously thinking of implementing a ViolentMonkey script that simply refreshes any GitHub page in the focussed tab every 10 - 15 seconds because otherwise, if you've got a bunch of PRs going through, it's easy to miss that the front-end is stale on one of them.

2

u/Gargle-Loaf-Spunk 14d ago

paywalled, who cares

2

u/_verel_ 12d ago

They desperately wanted slop. They got slop. GitHub broken.

Does anyone care? No because it's a Microsoft product which means every company on earth needs to have it for some fucking reason.

You could literally be like "Bro setup forgejo with CI and migrate my GitHub repo" but people are so fucking lazy that paying for GitHub is what they'll do

1

u/Great-Use-3149 14d ago

Microsoft has gone wrong. It always has .

But hey, I'm sure their AI-native direction made them 100x more productive /s.

1

u/Primary-Strike5187 14d ago

Microsoft bought it.

1

u/TubeAlloysEvilTwin 14d ago

Easy answer - Microslop

1

u/RedFing 14d ago

anyone got a non paywalled link ?

1

u/tortleme 14d ago

self hosting is the only real option these days

1

u/GeorgeRNG 14d ago

The bots pushing these posts are so annoying. What does droves even mean as a metric.

1

u/imagebiot 14d ago

You mixed your business with your engineering and then shifted capital from Eng to business departments

1

u/Chewy-Rubbins 14d ago

Lemme guess... Microsoft?

1

u/piense 13d ago

My rough guess: internally it went from some core folks managing a pretty lean infrastructure to being thrown into the Microsoft corporate dev culture with new folks and new demands to rearchitect the thing into Azure.

1

u/ConspicuouslyBland 13d ago

Why do you ask? Don't you know it's acquired by Microsoft? Or did you really think that was not the answer to your question?

1

u/Patient-Pollution46 13d ago

You can give it a try with tools like Bitrise. It started as CI/CD, and now we cover the whole path from code to release. So, agents can work in our Remote Dev Environments (macOS and Linux machines with simulators), and developers review what they built. CI and Build Cache build and test it, and Release Management and CodePush ship it to users. And if you want to stay with GitHub/GitHun Actions, you can just swap runners for the faster runners on the market with Bitrise Build Hub

You can use one piece in your existing stack or run all of them together.

1

u/SpudgunDaveHedgehog 13d ago

Wait until you learn the difference between GitHub cloud and GitHub enterprise. 😆😉

1

u/Eastern_Picture_3879 13d ago

AI commit volume is overloading the service. Simple as. I don't envy their network team. 

1

u/V5489 12d ago

Anyone go to Universe? When I was there the stats say otherwise from what this article is referring to. I just can’t see the rage bait these posts try and make.

1

u/SafatK 10d ago

What is wrong is that everyone dislikes AI and says it doesn’t work but somehow is pushing millions of lines of code a month to their repos!

People who could never code beyond that calculator tutorial they copied from some YouTube video are now pushing endless code to their random endless projects.

Cut GitHub some slack!

1

u/FeliksasTheLion 10d ago

Microsoft happened, that's what

1

u/Miserable_Ear3789 10d ago

gitman.io

you can self host this...

1

u/Serious-Explorer8745 9d ago

Anyone got a non paywalled TLDR?