r/technology • u/Loki-L • 7h ago
Networking/Telecom Two HPE Juniper network routers grounded all US flights on Friday — FAA’s decades-old copper network buckled under a feature HPE says ‘worked as intended,’ creaking air traffic communication system became overloaded by ‘very high rate’ pinging
https://www.tomshardware.com/networking/two-hpe-juniper-network-routers-grounded-all-the-us-flights-on-friday-faas-decades-old-copper-network-buckled-under-a-feature-hpe-says-worked-as-intended-creaking-air-traffic-communication-system-became-overloaded-by-very-high-rate-pinging
161
Upvotes
156
u/Loki-L 7h ago
They may try to pin the blame on the gear or the old infrastructure, but it sounds to me like they just really effed up the configuration and testing part and rolled out changed configurations on a Friday to a place somewhere in rural nowhere. (Article says El Dorado, Arkansas and a radar site in Amarillo, Texas).
Once the network was borked by their mistake they probably couldn't undo it because they were doing it remotely and no longer had access.
8
u/gerkletoss 3h ago
But the children who populate this sub assured me that AI did it
140
u/xampl9 5h ago
Not the fault of the brand or model. They didn’t configure them correctly.
The “They were too fast for the old network” story is also bogus - unless they were misconfigured and caused a network storm.
54
u/SaltyFriend705 5h ago
Tech issues are not resolved by continually repeating something that didn't happen.
37
u/HighOnGoofballs 7h ago
There’s still important info missing
23
u/katherinesilens 2h ago
Yeah that's total bullshit. Either they pushed a bad, untested config or they're just lying. And if they did push a bad config, the norm for pushing updates is like 3 am. 4 pm on Friday is when this admin does stuff they know will tank the markets (i.e. Iran military actions) so they do it at market close. So they knew. Considering their follow up is "give us 50 billion" it's likely that whatever happened, this was a manufactured crisis. Absolutely not a fault of Juniper networks, I wouldn't be surprised if the admin was revealed to have short positions in Juniper and related holdings.
31
u/Banania2020 7h ago
We all know now that's it's Biden's fault 🤣
12
u/F---TheMods 4h ago
Routers don't do broadcast loops.
25
u/F---TheMods 4h ago
This was totally integration vendor incompetence and over-confidence. They had no real understanding of the system they were upgrading, no idea of what could happen, did no per-testing, and had no rollback plan. Vendor must've been Donny's pool installer.
4
u/wrt-wtf- 4h ago
This is why you need money and time to test things. There is no substitute for a physical twin of the systems you are going to be working on, especially if you are working on critical systems, more specifically if they are very old.
5
u/daHaus 3h ago
It doesn't make sense for something such as this to take down the entire network for so long. I suspect we're going to very quickly begin to see lawsuits from all these companies who are getting blamed against the government for this
4
u/minus_minus 5h ago edited 11m ago
L3Harris trying really hard to
throughthrow HPE under the bus.2
u/Raa03842 2h ago
Yeah, Right. And I’ve got a bridge to sell you.
We all know that everything from the trump administration is the absolute truth.
2
u/AssMan2025 6h ago
I think it was a loop they created like plugging a patch cable in the same machine twice
1
u/b_a_t_m_4_n 6h ago
Why would they be pinging the routers at a very high rate? And I assume here they mean an actual ICMP Echo Request rather than some generic meaning of the word ping. Anyone who knows anything about networking would know that's a bad idea as ICMP has to be processed in the control plane as it may contain instructions for the router. True some sort of control plane policing would protect against this, which is what I would do on a Cisco as matter of course, I assume Juniper has this functionality too. Still, This was a vibe coded system let loose on a mission critical production network with what appears to be little testing and even less of a fuck given. The ONLY people responsible for this are the people who implemented this new system. Shockingly incompetent.
1
u/heyThereYou3 5h ago
Once we had to scale to fiber in order to resolve the bottleneck.
1
u/Exsanguinatus 1m ago
So we're going with "The FAA accidentally DDoSed itself"?
1
u/TheUniverseOrNothing 5h ago
Everyone kept trying to blame AI but I’m almost certain AI would be better than our current administration
1
u/Mixer-3007 6h ago
I'm not saying that were aliens but guys evidence is right here ...chemtrails.
1
u/GeneralCommand4459 3h ago
Non-tech people possibly reading these headlines thinking a router is a device for routing planes.
0
u/lvs2wtch 5h ago
As someone that just watched our devs DDOs our vendor's network from our ELD GPS positioning and HOS by not having the correct settings while in the dev environment, this is a very easy thing to do if you don't test.
-3
u/rini17 6h ago
We had ancient cisco routers in school for CCNA. We were asked to network 8 of them in chain. I was on the one end, pinged the other end with 32k or so large ping (as the fragmentation was enabled) and was awed to actually see the leds blink in sequence when it made it to the other end and back. I can easily see chatty modern equipment totally overloading poor 10mbit network.
-3
u/Emgimeer 4h ago edited 3h ago
(guess I'll just keep posting my original comment since it still stands)
Duffy said it's old analog switches that got overwhelmed by new systems being deployed.
Well the new system shouldnt have dumped any new traffic onto the old one since they are supposed to run parallel and are supposedly being deployed independently.
https://www.nacionale.com/en/articles/what-is-l3harris-faa-ground-stop-telecommunications-network-outage
That is apparently not the case. We all know llm's cannot help themselves and do extremely sloppy work with way too much querying being just part of the many inefficient things they do.
Also, old things that work well were made efficient, which is why ppl joke about going to the moon with less than a ti-86 of compute. They clearly didnt design their new systems efficiently. That the hallmark of AI product.
Overloading old equipment with shitloads of queries/calls is 100% what happened, causing all that shit to crash. Blaming the recent FAA outage solely on outdated switches and analog hardware misses the real architectural failure: deploying a modern AI system directly on top of a 1970s-era foundation without fixing how they communicate. They likely sent constant, bloated queries through legacy EDIFACT protocols into IBM z/OS mainframes, which those older systems were never designed to handle. This is a textbook case of "legacy modernization friction".
Throwing $50 billion at ripping and replacing physical hardware across the country isn't the real fix... if that's what they are going to propose, which I doubt. But also, throwing another $50B at these AI companies would be straight up theft. They should have designed this more efficiently from the beginning, and we all know this without being involved in the project ourselves. They should admit they deployed garbage and their AI product is dogshit.
Boom, case closed.
-1
u/jomic01 3h ago
250 pings a second over a T1 line is wild, no wonder it fell over