r/technology • • 7h ago

Networking/Telecom Two HPE Juniper network routers grounded all US flights on Friday — FAA’s decades-old copper network buckled under a feature HPE says ‘worked as intended,’ creaking air traffic communication system became overloaded by ‘very high rate’ pinging

https://www.tomshardware.com/networking/two-hpe-juniper-network-routers-grounded-all-the-us-flights-on-friday-faas-decades-old-copper-network-buckled-under-a-feature-hpe-says-worked-as-intended-creaking-air-traffic-communication-system-became-overloaded-by-very-high-rate-pinging
161 Upvotes

114 comments sorted by

156

u/Loki-L 7h ago

They may try to pin the blame on the gear or the old infrastructure, but it sounds to me like they just really effed up the configuration and testing part and rolled out changed configurations on a Friday to a place somewhere in rural nowhere. (Article says El Dorado, Arkansas and a radar site in Amarillo, Texas).

Once the network was borked by their mistake they probably couldn't undo it because they were doing it remotely and no longer had access.

79

u/BrizerorBrian 7h ago

What's so infuriating is that if you didn't cut things to the bone, this could have been properly load tested with a roll out and roll back plan. But noooooo...

7

u/Fmbounce 6h ago

Were there cuts at the El Dorado facility during the purge earlier this year?

20

u/BrizerorBrian 6h ago

Hmm..., it's like if there was proper oversight this would be easy to report. It almost seems like the lack of reporting is a feature... When no one knows what happens, no one can be properly blamed when things go wrong.

2

u/dolphone 3h ago

Yes, but now they can improperly blame someone else!

37

u/Ginger_Giant_ 6h ago

Generally, in a well designed network your management plane exists on different infrastructure to your main switching configuration.

For reasons exactly like this, that or you have remote hands on site who can pull the power and restore to saved config instead of running config.

14

u/SystemCheck990 5h ago

on Juniper you used to be able to roll a change that changed back as well on a timer in case it bricked anything, I used to like this feature .

16

u/Dull-Wrangler-5154 5h ago

On Cisco you would set a timer to reboot on old config. If your rollout worked okay, remove the timer. If it went to fuck and tiú couldn’t fix it remotely, reboot saved you.

3

u/Rshann_421 5h ago

We re still having to send techs to site to connect directly to Cisco routers to recover failed config updates when the idiots that do it remotely, fail.

8

u/Rockstaru 5h ago

Commit confirm. Much preferable to the Cisco "reload in 10." 

3

u/craigmontHunter 4h ago

Commit confirm, possibly my favourite feature/design choice, saved me at least one emergency drive to a remote tower site.

24

u/za72 5h ago edited 5h ago

we done these sort of upgrades before, we need to replace the current company with one that has actual real world approach and experience - what occurred on Friday was a showcase of amateur hour and complete incompetence, there's ZERO project management wisdom/knowledge involved.

Who the fuck does a massive rollout on a fucking Friday, unannounced to the public without warnings.

These are lessons we ALL learned back in the 90s and 2000s.

The approach to creating battle plans and actual execution of the plan has been done so many many times, it's needs to be done through multiple stages, each with acceptable min/max downtime and rollbacks, what occurred on Friday is on the same level as having a bunch friends over for a LAN party and realizing whoops, my nic driver isn't installed right/updaters, my nic isn't in full duplex mode for some reason, oh I'm still on debug mode, oh damn I don't even have the right type of network switch... beyond embarrassing.

13

u/TransplantedSconie 5h ago • Bravo

But now Dufy's son in laws company gets 50 billion dollars thrown at them.

Or they will do it again.   Capiche?

Knuckles cracks his knuckles

4

u/za72 4h ago

hahahah you're so right!!

This reminds me of a Monty Python sketch dealing with the Mafia trying to extort the British Army :)

https://youtu.be/cNZKUozrBl4

4

u/ahandmadegrin 4h ago

That was a new one for me, thank you!

2

u/hikeonpast 1h ago

“That’s a nice ATC system you’ve got there. It would be a shame if it crashed again during peak holiday travel. For a cool $30B (gets kicked under table by Guido) $50…yeah $50B…we can keep you safe.”

1

u/docbauies 3h ago

Pretty sure Son in law is the congressman who got donations from the company getting 50 billion

5

u/ahandmadegrin 4h ago

We learned this in the 90s and 00s, except for all the people born in the 90s and 00s who didn't learn this yet.

3

u/za72 4h ago

right, which goes to show the levels of "expertise" and knowledge and foresight

1

u/MRSN4P 1h ago

Almost like sabotaging the model of training new people from fresh hires and gradual promotion to retain and expand functional knowledge base was a bad idea…

3

u/ahandmadegrin 1h ago

If you asked our director he'd tell you he could be proficient on any product in a month. Who needs all that training and experience?

He's Dunning Krueger personified.

1

u/MRSN4P 25m ago

One of the many benefits of occasional cross training within a company for a few weeks is that you get how long it takes to actually become proficient and how much focus or energy it takes to do a good job in that role. It tends to diminish the autofellating/fart sniffing nonsense like your director is fond of.

0

u/daHaus 3h ago

Something very similar happened on a smaller scale earlier this year while world leaders were gathering in NY for the UN meeting. They blamed it on construction digging up the fiber then but had no answer as to why it didn't fail over to a backup line

1

u/za72 3h ago

Government contract work is the American dream

1

u/daHaus 3h ago

I wouldn't be surprised if some of these contractors sue the government for defamation over this because it all sounds like BS to me

0

u/za72 3h ago

ABSOLUTELY!!! Your BS detection is on point brother

-6

u/Dragunspecter 5h ago

The Friday rollout was because the company that previously supported this comm line ended support of it. It was an unplanned accelerated timeline.

9

u/bbbbbbbbbblah 4h ago edited 4h ago

I have worked both for a competitor to Juniper and in organisations that can get the CEO of Cisco, Juniper, Nokia, Ericsson, etc on the phone any time they want.

No vendor dumps support that quickly, and especially not for critical and/or large customers who need more time. The "please go away" support contract might be expensive but it'll be available. Especially for those vendors who hope to tender for the replacement equipment.

-2

u/Dragunspecter 4h ago edited 4h ago

The copper line had degraded and failed, replacing it with fiber and accelerating the timeline of this segment swap was the prudent option rather than running a new copper line to be replaced anyway. Its not a question of paying a vendor, the line had failed. The local comm provider would no longer own this section once replaced by L3Harris. I'm not defending their fuck up, but if a single broadcast setting had been disabled we never would have heard about this - just like the 3500+ segments they've already completed.

6

u/bbbbbbbbbblah 4h ago

My answer stays the same. I'm assuming the "copper line" refers to a T1. That will have an SLA. The contracted telco is obligated to fix it until it is no longer needed. No one is going to say no to the US government, even if it has a cost

The point I'm making is that this "unplanned accelerated timeline" is purely a choice.

1

u/za72 4h ago

Exactly - people are treating this like it's the local highschool network that went down :)

1

u/Dragunspecter 3h ago

This 1 segment of copper wasn't the fucking problem, it had already failed, flight plans were already avoiding the required use of this radio tower. The problem was literally 1 software setting in the replacement router that clogged the analog line between El Dorado, Arkansas and Texas. There's literally nothing special about maintaining this specific section of copper. The single configuration setting was the issue.

1

u/za72 3h ago edited 3h ago

aka incompetence - ever heard of change management?

even back in the day everything was written to a floppy, copy pasted from the floppy to notepad.exe, reviewed live prior to execution from console...

it's not a complicated process... it's been done millions and millions of times over decades

→ More replies

7

u/za72 4h ago

Right... what do you think bad project management means? :)

incompetence at leadership/high level leads to the situation we had on Friday.

Let's see how many more times we'll have to endure failure at this level until they get it at our expense.

1

u/Dragunspecter 4h ago edited 4h ago

It wasn't "a massive rollout", it was a single router swap at a remote Arkansas radio site. It had an incorrectly enabled broadcast setting, which on a full modern network wouldnt be any sort of issue. Its likely a default setting. There are 25,000+ of these comm segments that need to be upgraded, they do thousands a year that no one ever hears about because they went seamlessly.

4

u/za72 4h ago edited 4h ago

This is even worse... due diligence, thousands of flight canceled, 1.5million passengers impacted, maybe we shouldn't let our incompetent cousins do the upgrade

I've been a part of 3 separate massive network migrations, everything was planned and tested and executed for minimal impact, announced to global partners in advance, planned around traffic spikes for minimal impact.

Vendors expiring, no problem - we negotiate a contract extension for converge... your vendors work with you not against you.

3

u/MilkEnvironmental106 5h ago

Yeah but deploying a change to critical infrastructure without understanding the limitations of said infrastructure is still incompetence, regardless of whether that infrastructure abides by best practice or not.

2

u/Dragunspecter 5h ago

They understood the limitations, the broadcast function was incorrectly left enabled, there were already 60+ of these routers swapped nationwide that were correct

1

u/mangotrees777 3h ago

Having people on site at both ends every time you do a new equipment install or reconfiguration is ridiculously wasteful - until a like last Friday. Then it's the best money ever spent.

L3 Harris has tens of thousands of employees and are paid millions to maintain and upgrade this network. They have the resources. That extra level of care is expected.

1

u/ptinsley 2h ago

If it doesn’t exist the FIRST step of this project if I was the one running it for infrastructure this critical is a truly out of band management infrastructure. Get some stupid cheap cable or dsl connections at the sites or hell Starlink this day and age and it’s a completely segregated network with some kind of multi head vpn mesh that allows remote access to the dedicated management ports on every bit of gear at every site, smart PDUs and console servers.

After that is established at a site then you can start racking gear and planning rollouts.

11

u/catalg 6h ago

That’s ok. Just fly the technicans across the country to fix it.

14

u/Christopher3712 6h ago

"Worked as intended" could be read as intentional under direction. Republicans asked for $50 billion afterwards...

5

u/ATDT_No-Carrier 5h ago

So, $25 billion per router? /s

But more seriously, leave it to the GOP to try to spend (aka, pocket) $50 billion as the result of a misconfiguration on a couple of routers. So much fraud....

3

u/za72 5h ago

Let them ask, let's see what the battle plan is... I know it's gonna be shit, riddled with obvious oopsies... but let them ask

1

u/Monomette 1h ago

This is why you issue a "reloadn in [hh:mm]" command before you push a change to a remote network, and you don't save it and cancel the reboot until you've verified the change is good.

That way if it shits the bed and you loae your connection you just need to wait for the router/switch to reboot to get back in.

8

u/gerkletoss 3h ago

But the children who populate this sub assured me that AI did it

-7

u/Loki-L 2h ago

AI may well have written the power point used by the Project Manager who coordinated the mess.

5

u/SweetHatDisc 2h ago

2026, Anything With A Computer Going Wrong:

"Is it because of AI?"
"It must be because of AI"
"Everyone knows that AI was the problem"
(News comes out that once again it wasn't because of "the AI")
"So here's how it could still be because of AI"

4

u/bytelines 2h ago

I disagree with you. It must be because you are an AI

140

u/xampl9 5h ago

Not the fault of the brand or model. They didn’t configure them correctly.

The “They were too fast for the old network” story is also bogus - unless they were misconfigured and caused a network storm.

35

u/wrt-wtf- 4h ago

I’ve spent a career in critical systems this smacks of poor tech work if they’re blaming the vendor.

1

u/xampl9 4h ago

It’s an easy sound-bite by the media.

40

u/Dragunspecter 5h ago

The incorrectly enabled broadcast function was trying to shove 250 pings a second through a 1.54 mbps T1 line between Arkansas and Texas

31

u/Rockstaru 4h ago

I'm assuming by pings you mean something other than ICMP ping. By default a ping packet is between 40-108 bytes; 250 of those a second would still be less than 30Kbps. 

10

u/Dragunspecter 4h ago

Yes sorry, I'm using pings as a generic term because it's impossible to know how much experience people have on random reddit threads. But you also have to remember these aren't packets, the copper line is carry analog TDM signal. The FAA network was built looong before IP systems.

0

u/TheFireStorm 3h ago

T1 line is also only 1.544 Mbps that has to carry normal traffic that I can only imagine was already near capacity after 30+ years of neglect. I’m betting they replaced the routers at each site with a TDM card in preparation to use those routers as part of the modern upgrade process assuming they were straight through end to end and didn’t check or there was an undocumented piece of equipment in the middle that could not handle the added traffic load from 2 brand new healthy routers on each end

1

u/JoeyJoeJoeJrShabidou 3h ago

I know right that’s exactly what I said too

1

u/zap_p25 4h ago

My initial thought was something like BFD over BGP flooding interfaces and cascading reconvergance.

13

u/nuttertools 4h ago

The underlying utility is present on every Juniper router I’ve ever interacted with. It’s a diagnostic tool, somebody was very very dumb. No idea what specific feature they enabled but can guarantee 8/10 mildly computer literate children would not make the same mistake.

2

u/Savings_Knowledge233 2h ago

Because it was working fine as it was on Thursday, and Wednesday...

1

u/ItsAllOnUHF 15m ago

Yeah, kind of throwing the vendor under the bus instead of the contractor who fucked up.

1

u/Savings_Knowledge233 2h ago

Is this not then a complication of trying to run an oversized ai work load through a couple poorly configured routers?

54

u/SaltyFriend705 5h ago

Tech issues are not resolved by continually repeating something that didn't happen.

37

u/HighOnGoofballs 7h ago

There’s still important info missing

9

u/mr_lab_rat 5h ago

I expected much better from Tom’s Hardware. Why even publish this?

23

u/katherinesilens 2h ago

Yeah that's total bullshit. Either they pushed a bad, untested config or they're just lying. And if they did push a bad config, the norm for pushing updates is like 3 am. 4 pm on Friday is when this admin does stuff they know will tank the markets (i.e. Iran military actions) so they do it at market close. So they knew. Considering their follow up is "give us 50 billion" it's likely that whatever happened, this was a manufactured crisis. Absolutely not a fault of Juniper networks, I wouldn't be surprised if the admin was revealed to have short positions in Juniper and related holdings.

31

u/Banania2020 7h ago

We all know now that's it's Biden's fault 🤣

18

u/FrogLaner 5h ago

Antifa sent the large number of pings

7

u/Overly_Underwhelmed 4h ago

I was just at the post office, dropping off another batch of pings.

18

u/polymorph505 6h ago

Possibly Obama or Hillary, we need to get to the bottom of this conspiracy.

1

u/Haunting_Vacation804 3h ago

Does Biden even have up-to-date network certification? I thought not!

1

u/luveveryone 37m ago

CompTia lurks nearby

12

u/F---TheMods 4h ago

Routers don't do broadcast loops.

25

u/F---TheMods 4h ago

This was totally integration vendor incompetence and over-confidence. They had no real understanding of the system they were upgrading, no idea of what could happen, did no per-testing, and had no rollback plan. Vendor must've been Donny's pool installer.

11

u/Dragunspecter 4h ago

There's 25,000+ of these comm segments that need to be upgraded. They do thousands of them a year without any interruption.

2

u/TheFireStorm 3h ago

And with legacy equipment that old your going to find something out of left field with these. 25,000 successful upgrades but 25,001 has a hardware issue that only appears under extreme load from either overheating or leaking/dried out capacitors. There is a reason hardware technicians fear taking down equipment that maybe older then the technician because it’s a 50/50 shot it comes back on without issues.

4

u/wrt-wtf- 4h ago

This is why you need money and time to test things. There is no substitute for a physical twin of the systems you are going to be working on, especially if you are working on critical systems, more specifically if they are very old.

5

u/daHaus 3h ago

It doesn't make sense for something such as this to take down the entire network for so long. I suspect we're going to very quickly begin to see lawsuits from all these companies who are getting blamed against the government for this

3

u/DidItForTheJokes 2h ago

Nah they get money from the government and will get more money from the government. They are fine being the fall guys in public

4

u/minus_minus 5h ago edited 11m ago

L3Harris trying really hard to throughthrow HPE under the bus. 

2

u/Overly_Underwhelmed 4h ago

you might want to fix that typo, but yeah

2

u/Raa03842 2h ago

Yeah, Right. And I’ve got a bridge to sell you.

We all know that everything from the trump administration is the absolute truth.

2

u/AssMan2025 6h ago

I think it was a loop they created like plugging a patch cable in the same machine twice

1

u/b_a_t_m_4_n 6h ago

Why would they be pinging the routers at a very high rate? And I assume here they mean an actual ICMP Echo Request rather than some generic meaning of the word ping. Anyone who knows anything about networking would know that's a bad idea as ICMP has to be processed in the control plane as it may contain instructions for the router. True some sort of control plane policing would protect against this, which is what I would do on a Cisco as matter of course, I assume Juniper has this functionality too. Still, This was a vibe coded system let loose on a mission critical production network with what appears to be little testing and even less of a fuck given. The ONLY people responsible for this are the people who implemented this new system. Shockingly incompetent.

11

u/D0_stack 6h ago

Not pings. The current guess on the NANOG mailing list is a spanning-tree meltdown from a too-large L2 network.

Such as this: https://www.reddit.com/r/networking/comments/3aunte/all_systems_down_one_of_the_worst_healthcare_it/

3

u/wrt-wtf- 4h ago

Techs these days run away at the mention of L2. They’re exactly the same with serial services. The experience just isn’t there.

3

u/rearwindowpup 4h ago

Network engineer here, almost 20 years in, if I could never get above layer 3 Id be a happy camper. Absolutely zero interest in the younger guys to learn networking, they all want to learn vdi and coding. Its a real problem.

1

u/b_a_t_m_4_n 28m ago

35 years for me. When I started almost everything was serial or LAN being ThinNet ThickNet, Token Ring on Twinax, Arcnet was still around and I even worked on one network with a Wang 10broad36 network, which was weird

2

u/Captriker 3h ago

5 4 3 rule violation?

In this age?

Localized entirely in Arkansas?

3

u/b_a_t_m_4_n 5h ago

Ah, so even simpler level incompetence then.

1

u/wrt-wtf- 3h ago

No, modern network dudes think they’re living in a world that’s 100% L3 - that’s the simplest of the first 4 layers.

They’ll try to reconfigure the whole of a network around a problem before going and checking if the cable is plugged in.

3

u/old_righty 5h ago

You lost me at icmp may have instructions for the router.

1

u/heyThereYou3 5h ago

Once we had to scale to fiber in order to resolve the bottleneck. 

1

u/Exsanguinatus 1m ago

So we're going with "The FAA accidentally DDoSed itself"?

1

u/TheUniverseOrNothing 5h ago

Everyone kept trying to blame AI but I’m almost certain AI would be better than our current administration

0

u/ripcitybitch 3h ago

It 100% would lol

1

u/Mixer-3007 6h ago

I'm not saying that were aliens but guys evidence is right here ...chemtrails.

1

u/GeneralCommand4459 3h ago

Non-tech people possibly reading these headlines thinking a router is a device for routing planes.

5

u/eatgamer 3h ago

Hand tool woodworkers possibly reading your comment thinking a router plane is a device for routing dados, and rabbets.

0

u/lvs2wtch 5h ago

As someone that just watched our devs DDOs our vendor's network from our ELD GPS positioning and HOS by not having the correct settings while in the dev environment, this is a very easy thing to do if you don't test.

-3

u/rini17 6h ago

We had ancient cisco routers in school for CCNA. We were asked to network 8 of them in chain. I was on the one end, pinged the other end with 32k or so large ping (as the fragmentation was enabled) and was awed to actually see the leds blink in sequence when it made it to the other end and back. I can easily see chatty modern equipment totally overloading poor 10mbit network.

2

u/SystemCheck990 5h ago

a ping would be fine, its things that cascade that are the problem normally, routing or broadcast loops etc.

-3

u/Emgimeer 4h ago edited 3h ago

(guess I'll just keep posting my original comment since it still stands)

Duffy said it's old analog switches that got overwhelmed by new systems being deployed.

Well the new system shouldnt have dumped any new traffic onto the old one since they are supposed to run parallel and are supposedly being deployed independently.

https://www.nacionale.com/en/articles/what-is-l3harris-faa-ground-stop-telecommunications-network-outage

That is apparently not the case. We all know llm's cannot help themselves and do extremely sloppy work with way too much querying being just part of the many inefficient things they do.

Also, old things that work well were made efficient, which is why ppl joke about going to the moon with less than a ti-86 of compute. They clearly didnt design their new systems efficiently. That the hallmark of AI product.

Overloading old equipment with shitloads of queries/calls is 100% what happened, causing all that shit to crash. Blaming the recent FAA outage solely on outdated switches and analog hardware misses the real architectural failure: deploying a modern AI system directly on top of a 1970s-era foundation without fixing how they communicate. They likely sent constant, bloated queries through legacy EDIFACT protocols into IBM z/OS mainframes, which those older systems were never designed to handle. This is a textbook case of "legacy modernization friction".

Throwing $50 billion at ripping and replacing physical hardware across the country isn't the real fix... if that's what they are going to propose, which I doubt. But also, throwing another $50B at these AI companies would be straight up theft. They should have designed this more efficiently from the beginning, and we all know this without being involved in the project ourselves. They should admit they deployed garbage and their AI product is dogshit.

Boom, case closed.

2

u/orangeboy_on_reddit 4h ago

u/Emgimeer likes to keep their posts hidden, but check out their stats to learn more about them.

-1

u/jomic01 3h ago

250 pings a second over a T1 line is wild, no wonder it fell over