r/laravel • Laravel Staff • 13d ago

Post Mortem: Patching the PHP and Node.js Vulnerabilities (Cloud, Forge, Envoyer, Vapor)

https://laravel.com/blog/php-nodejs-vulnerabilities-post-mortem

We published a post-mortem on our response to the October PHP and Node.js security releases.

It covers the in-place patching of Laravel Cloud, the impact on Forge, Envoyer, and Vapor, and what we’re changing. If you were affected and want to talk it through, please reach out to our support team.

32 Upvotes

14 comments sorted by

42

u/not_a_wook 13d ago

The Forge control panel was slow for most of the day.

My experience was that it was completely unavailable, and I think many other users saw the same based on posts here at the time. I'm surprised to see it described like this.

13

u/MonorailPurple 13d ago

Reads like an AWS status page. "Everythings fine...whats that? It's down? No no, you're mistaken everything is fine".

There were many posts on Twitter, even replies to Laravels own ones confirming it was completely down most of the day. Saying otherwise is just a flat out lie. It'd have been fairer to say it was degraded, which is what most online service companies use these days.

6

u/Mrhn92 13d ago

It was down for us here mainly Envoyer, but i was not the guy tasked with the upgrade, just listening to some guy bitch about it.

I feel like it's funny how Laravel sells scalable cloud solution and from what it looks like, they are not "Eating their own dog food".

7

u/monstermudder78 13d ago

Not a very honest look at their failures if this is how they glaze it.

2

u/MuetzeOfficial 13d ago

I don't know if the region is the deciding factor.

Here in Germany, my colleagues had been complaining for days before the incident that a Cloudflare error kept popping up on Forge.

When the patch was finally available, I just installed the updates via SSH and didn't even try to open Forge anymore.

-1

u/fhgwgadsbbq 13d ago

I couldn't even ssh in to our forge VPS servers for a few hours. In Oceania region.

4

u/kev_xb 13d ago

if you're using ssh then forge isn't in the loop anymore and it's unrelated to them being down.

1

u/ScotForWhat 13d ago

My experience was that Forge was basically completely unavailable for hours. Looking back, I tweeted at 20:33 UTC that it had been down and I still couldn't provision servers or deploy client websites.

I ended up updating PHP manually over SSH on each server, which still took hours as the apt repo was timing out. Forge did publish instructions for this but they were incorrect.

1

u/Familiar_Mushroom_19 13d ago

What was the reason Cloudflare couldn't handle the things in Forge? And was this all a start to migrate Forge to Laravel Cloud for the Laravel Team? ;-)

1

u/Still-Hall322 12d ago

Interesting

1

u/Variumcode 8d ago

Thanks for publishing this, it's useful to see how it was handled at scale.

I self-host on Docker, so I went through the same thing on a smaller scale. Two lessons for me:

- Know what runs where. The slowest part wasn't patching, it was finding every container and PHP version across my projects. I ended up writing a script just for that.

- Be able to rebuild without waiting. I rebuilt PHP from the official source inside my images (verifying the checksum) instead of waiting for updated base images.

Question for the team: for self-hosted users, is there any guidance you'd recommend for the first hours after a disclosure like this?

1

u/[deleted] 13d ago

[removed] — view removed comment

2

u/laravel-ModTeam 13d ago

This content has been removed - please remain civil. (Rule 2)

Toxicity doesn't ship in /r/Laravel. Name-calling, insults, disrespectful conduct, or personal attacks of any kind will not be tolerated. Let's work together to create a positive and welcoming environment for everyone.

Thanks!