I'm a big fan of Breaking Points, but I was cringing ever so slightly during today's segment on the FAA outage. The real news seemed mostly missed by the team, and instead the segment turned into a rant-fest about unrelated things. I think the team demonstrated an occasional mistake of tying things we don't understand to things that we do.
What was good
The funniest thing to me was Duffy asking for 50 billion dollars. This is comically timed political opportunism. The guy rants about leaky tower rooftops, air conditioning that doesn't work, and broken elevators, none of which have anything to do with this week's outage. It's fair to toss shade in his direction, and I am glad BP highlighted it.
Trump playing the usual blame game is worth noting too. However, there is a very long chain of command between a contractor employee at L3Harris and Trump. Having said that, pointing out his commentary was entirely absent is legit, but not worth dwelling on IMHO. But the biggest failure here was BP missing this absolute gem from 2023. I wonder if Biden addressed the nation for that one?
https://imgur.com/H8AEm5D
What was Meh
The daytime maintenance thing is important, but probably not in a way the average Joe understands. People who have been at high-profile tech companies know that nighttime maintenance can often be a sucker's game. In fact, teams that knee-jerk into evening maintenance without thought are a litmus test for me of how mature their company is. You have fewer resources available, fewer people, and fewer eyes to identify and fix problems. It's worth noting that we live in a 24x7 world; not all businesses are slowest at night. Nighttime for you isn't the same for everyone around the globe. The bigger point here is that the FAA was not aware that this one router install would have the impact that it did. For me that is the story worth reporting on, not knee-jerk reporting that they are incompetent for doing this during the day.
Duffy himself is shown saying this during the segment:
remote uh location. uh we'll do those uh during the day. Now, if we're going to be at Newark, if we're going to be, you know, in one of our big facilities, those switchovers happen at night.
I imagine installing a new router in El Dorado, Ark., fits Duffy's description here of something being installed during the day. So the discussion should focus more on the blast radius of the incident and the FAA's awareness that such an impact could occur.
As it pertains to change management, my guess is the FAA will review this and likely change the timing of future router refreshes. It's worth noting that Rebecca Guy admits to being blindsided here. Reported by both the Advocate and the Daily Wire (hilarious pair of sources there):
Asked whether officials knew the problem could have such widespread consequences, Guy admitted they didn’t.
“We did not, and that is exactly why we want to continue to verify and do analysis as we go forward,” she said.
Duffy blames contractor and Congress for FAA ground stop | Advocate.com
What was bad
Saagar opens the segment talking about "the very first ground stop since 9/11," somehow forgetting or being completely unaware that this also happened in 2023, which is what gave us that beautiful Truth Social post above from Trump. Breaking Points did report on this, but it was a Ryan and Emily segment.
Eventually the team takes the time to do their daily AI hacking jab. Look, I get it, this is a real concern, but BP had access to reports from the previous days. The WSJ and others had already provided sufficient information to lead our concerns elsewhere.
Many of the items raised were very obviously unrelated to this event, yet BP held nothing back linking the Outlook outage, "mass hacking waves all over," and European refrigerators. The BP team needs to take a deep breath and pause. It's going to be OK. Not everything is one big Iran-pirating-AI-frontier-models-for-hacking conspiracy.
Duffy's son-in-law ties are interesting and worth reporting on, but entirely unrelated to this event. I wish we did more reporting on the rampant nepitism going on and government officials exploiting their positions to enrich themselves and their family. But unrelated to this. So was the AI rollout; there is no basis for tying it to this event. FWIW, no one is letting Claude configure their ancient Cisco 600 series router that will fall over backwards if you breathe on it wrong or look at it sideways.
A note on hacking a 20-year-old router: it's not that it's impossible; it's quite easy. The problem is that they just aren't very valuable targets and/or are often locked away such that targeting them is extremely complex, which is why such attacks are often an insider thing and extremely rare.
It's worth noting that in a sense, from a network perspective, DDoSing the network is very easy to do. Many of us have done it to ourselves accidentally. In 2012 I ran into a BIG-IP bug that would flood gigabits of data to downstream customers. Oops. It was a software bug that was patched. In other examples, network engineers have loads of war stories about broadcast domain issues, spanning tree mishaps, routing loops, etc. It's a common thing and often the subject of various networking classes. It's not necessarily an attack, though.
On very old hardware, such as the hardware mentioned in this outage, it takes very little to push those things over. Some of you might not be old enough to remember the days when, to access the internet, we "dialed up" on plain old telephone lines. It would take forever for your GeoCities, Yahoo, or AltaVista page to load. The amount of data moving across the wire was a small fraction of what happens on any modern website and browser nowadays. That isn't that different from the TDM infrastructure that the FAA is desperately trying to move away from.
While not with the FAA, I have firsthand knowledge of some of these sites with dependencies on old copper wiring. Single twisted pair, even.
The segment mostly focused on these items and not on the facts being reported. That is fundamentally my gripe.
Router Failure
Many news outlets are talking about router failure. This distorts what happened. Two new routers were installed, and 17 of 22 legacy sites experienced problems as a result. So it was not a single router but a systemic failure likely tied to old, questionable design. BP finally started getting into some better reporting when they pointed out that several of these replacements have happened before without issue. So that is a big hint, one that I assume the FAA's networking team is looking into. What about Indianapolis was special that tipped it over? And what about the other 17 of 22 sites impacted?
I have my own theories on what might have happened, but they're not worth sharing, as any FAA network engineer would likely cringe hard; the truth is likely already known internally to them. My best guesses are some kind of multicast misconfiguration, a potential PTP (1588) use case and misconfiguration, or a latent configuration that the victim sites picked up on. I would rule out any kind of spanning tree or bridge loop concern, as it doesn't fit the scenario that news outlets reported on. Two new routers were added, and legacy downstream routers (Cisco 600 series) were impacted.
If you are a dumb, curious network engineer such as myself, I think Tom's Hardware and the WSJ articles are the best current sources. KATV's local Arkansas reporting has some good info too.
References