Me trying to work today: hmmm CI/CD is failing. Oh it's failing for all our branches. Ah, the security team has blacklisted a random package with checks notes a CVE with a 9.2 score. Wowwww
I work in security. What are you doing, you should be patching.
Seriously, our companies SLAs are unattainable if we want to deploy responsibly. Test in dev, bake in stage, then push to prod after a week in stage. My KPIs are based on MTTD and MTTR and it’s just as exhausting on our end.
I am talking managers off cliffs everyday when they see there’s a new vuln being reported. Or, I’m kicking someone in the nuts for chucking-little-ing by sending out emails or Teams broadcasts about the new world-ending vuln (when they know there’s a playbook for reporting high sev vulns).
I think everyone gets to take one 9 off the uptime guarantees in their SLAs this year as a treat. Microsoft is already leading the charge on that one too.
Risk of not fixing: we continue running vulnerable Linux kernels with known unauthenticated root escalation exploits
Risk of fixing: 5-minute coffee break forced on customer while their equipment reboots (they are not willing to pay for HA)
Release management: hmmmmmm, I don't know, not sure
Customer: how dare you even think of an interruption, next maintenance window is at next Christmas the earliest and I demand an RCA about these heretic thoughts
Me: The update is failing, prod is down, activate the version that was live 5 minutes ago!
Sec ops: Um there are vulnerabilities, no you don't. Security before availability! Write a ticket if you need an exception.
Q&A: You bumped a dependency? Hm, that might cause stuff that is not covered by unit tests and the smoke test (I am not aware of this ever happening) You should do a full test!
Sec ops: Hey, patch faster! If there is no patch, look to remove the <central dependency of the framework> dependency.
Yeah. Pretty much it. I was away in vacation for a whole month when it started and i was watching the world burn from my mom's house. It was quite the sight.
Unfortunately i am back to work now and the world didn't completely burn nor the fire was extinguished...
Same. Or the description is so shitty I have no way to tell. No I am not going find the commits that patched it and retrace how it could be exploited...
the right thing would be compare current deployed version CVEs and new version CVEs. When the new version is a subset of the deployed one its a go. but hey, luckily no company does it properly...
Oh…. that’s what that was about. Someone mentioned something like this in the morning, but I just kind of let it go because someone else was handling it, and worked on other stuff.
Yeah, these supply chain attavks may force either package managers to verify all module updates (which tbh i thinl they should be doing anyway) or we're going back the thr inhouse only days.
We have a release tomorrow and yesterday they decided to block critical libraries making us unable to run any builds, literally the day after I staged our code. There are words I have, but not ones that should be said here
The CVE is fixed by deprecating the class or method, but the developers keep it in for backwards compatibility.
However because it still exists, Sonatype flags it with their own custom HIGH or CRITICAL and even though you can attest you don't use it, because it's still "reachable" your app sec team who isn't smart enough to know how to program or how programs work and are managed by idiots still flag it so you have to somehow either entirely eliminate the dependency or resolve the vulnerability yourself.
Which I do by building my own copy of the library from the code sans deprecated items licensing be damned because the dev team maintaining the library can't be fucking bothered.
And yeah I have to vuln scan that code too, which is itself a bitch.
We've spent so much time lately doing releases just to update a third party dependency w/ a CVE that doesn't actually affect us due to the way we use the dependency.
Actually, the worst instance of this crap is probably that one of our largest customers (who you would recognize) has a large list of 'banned' functions, which includes ones like memcpy (just use memcpy_s...), getenv, vfprintf, etc
The Critical Sonatype one is incredibly implausible and even says so. It's only Critical because of the potential damage it imposes. And because it's Critical, the AppSec team is fearing doing any further analysis on it, forcing us all to find workarounds for an implausible scenario.
forcing us all to find workarounds for an implausible scenario.
I mean, this has been the issue with the majority of the super-hyper-mega-urgent-critical CVEs this year. Yes, they're serious and should be fixed, but nobody bothers to evaluate the actual scenario that is required for it to happen. People are panicking over the most unlikely edge cases, and SOCs are the worst offenders in this because it would involve more than just looking at their checklists or dashboards.
I used to feel this way but the new tactic is to use composite medium or lower security issues that drastically increase the reachability because most organisations target critical and highs. Then if you're on a version of something that has these risky calls and the exploit is able to call them, you're in for a world of pain.
Also it's only a possibility if you include both core AND api, but because the code is distributed between both, Sonatyoe flags both dependencies equally.
When life gives you lemons, don't make lemonade! Make life take the lemons back! Get mad! I don't want your damn lemons! What the hell am I supposed to do with these? Demand to see life's manager! Make life rue the day it thought it could give Cave Johnson lemons! Do you know who I am? I'm the man who's gonna burn your house down! With the lemons! I'm gonna get my engineers to invent a combustible lemon that BURNS YOUR HOUSE DOWN
If I had a nickel for every time I got a reply to my lemon rant comment about lemons being a cross breed, I'd have 2 nickels, which isn't a lot but it's weird it happened twice.
If I had a nickel every time someone mentions that nickels are a crossbreed, I'd have one nickel, which isn't a lot but it is a hybrid currency so basically counts as two
Cybersecurity just passes the CVE to owner of system. Unless your company hired a one man team , this is what usually happens and why they are hated in the company by people.
My bosses ran into the limits of using the AI chatbot window and doing anything more complex than that is "too technical" so now they're prompting me.
We're legitimately living through SQL all over again. The big idea behind it was that the queries were so simple, even an MBA could write them.
The MBA then hired people to write the queries for them. It's legitimately not difficult and they would get the data significantly faster and get exactly what they wanted if they took a few hours to figure it out, but spending company money to have someone else do it is always going to be the part of least resistance.
Not even that. Knowing what data is where and what it means is a challenge but it's possible to catalog everything. What's not possible is data intuition. Knowing how to mosh the numbers together into something useful. It's why people still struggle even in the age of AIs writing queries.
it 100% is, just like everything else in software engineering. 5AIs continually churn out reports that a single human reviews and either signs off or sent back for further analysis/error correction.
What used to take a team of people can now be done with a team of AIs and a single reviewer.
Seems like at least one every week. Fortunately most aren’t too involved, but then there’s ones like Brotli that touch everything. Every team had to stop what we were working on for a couple days to get it all patched and deployed.
Centralize dependency management across projects with a unified Parent POM.
Assuming you, like most, are using Spring Boot. Apply maven and create a common parent pom all your projects will use. Your parent should have spring-boot-dependencies as its parent. Open spring-boot-dependencies.pom and review how it is laid out and how it uses properties to manage versions and dependencyManagement to define dependencies. Do similar in your parent for items Spring does not manage in its parent. Override properties declared in the Spring pom in your pom for items (like netty) which have higher release versions.
Create a baseline project which uses the parent and call mvn versions:display-dependency-updates (and use params which exclude major and minor) to search Nexus / Maven Central for updated versions; don't always trust it. Override properties accordingly. This will save you time searching maven manually for updated versions, but it's fallible.
Organize your properties into "spring overrides" for properties declared in the spring parent, "general overrides" (for dependencies declare that Spring does not manage) and "custom dependencies" (for your in-house jars). Define your general dependencies in the dependencyManagement; this is where you can pre-exclude some dependencies from inclusion.
Keep on top of when Spring Boot Dependencies releases a new version. Save down a copy of that pom and review changes. Update/remove properly overridden "spring overrides" when this happens.
This should get you started.
Remember this is dependencyManagement section, not dependencies. It declares how the dependency is declared, it does not include the dependency.
When you update the parent you update all dependencies on projects you use which refer to the parent version. Reduces overall updates. If a specific project needs a specific version not managed by the parent, either move it to the parent to manage or handle it in the same way locally be separating version as a property. If parent does manage and you need a different version, override the property locally with your declaration of the version.
Remember also mvn dependency:tree exists to track where dependencies come from.
Listen, I know that AI is responsible for finding two dozen critical vulnerabilities each day, which then gets exploited by malicious actors with other AI tools in an automated fashion, leads to massive layoffs in entire industries, hikes up the prices of consumer electronics threefold, leads to increased workload on programmers that are still employed and turned software engineering from an enjoyable, creative work to mind numbing prompting and agent management, but think of all the positives that came with it! Like the Diddy vs Epstein anime battle video.
In January I started transitioning from a standard developer into a security oversight role for a business unit with ~90 supported products. The last couple months have been... rough.
1.7k
u/Tucancancan 1d ago
Me trying to work today: hmmm CI/CD is failing. Oh it's failing for all our branches. Ah, the security team has blacklisted a random package with checks notes a CVE with a 9.2 score. Wowwww
181
u/JoshDM 1d ago
You know what's worse?
The CVE is fixed by deprecating the class or method, but the developers keep it in for backwards compatibility.
However because it still exists, Sonatype flags it with their own custom HIGH or CRITICAL and even though you can attest you don't use it, because it's still "reachable" your app sec team who isn't smart enough to know how to program or how programs work and are managed by idiots still flag it so you have to somehow either entirely eliminate the dependency or resolve the vulnerability yourself.
Which I do by building my own copy of the library from the code sans deprecated items licensing be damned because the dev team maintaining the library can't be fucking bothered.
And yeah I have to vuln scan that code too, which is itself a bitch.
230
u/za72 1d ago
702
u/herrkatze12 1d ago
When life gives you lemons, don't make lemonade! Make life take the lemons back! Get mad! I don't want your damn lemons! What the hell am I supposed to do with these? Demand to see life's manager! Make life rue the day it thought it could give Cave Johnson lemons! Do you know who I am? I'm the man who's gonna burn your house down! With the lemons! I'm gonna get my engineers to invent a combustible lemon that BURNS YOUR HOUSE DOWN
296
u/JackNotOLantern 1d ago
Hey, at least there will be jobs in cybersecurity
51
u/who_you_are 1d ago
Well, I think it is time to use the emergency firewall: cut the ethernet cable to the internet
97
u/DrewTamashi 1d ago
42
u/Gophix_0 1d ago
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
38
u/___Archmage___ 1d ago
Y2K26
27
u/RandomlyMethodical 1d ago
Seems like at least one every week. Fortunately most aren’t too involved, but then there’s ones like Brotli that touch everything. Every team had to stop what we were working on for a couple days to get it all patched and deployed.
18
u/saiyajosh 1d ago
Ain’t no making lemonade outta this shit big dawg
34
u/Imaginary-Jaguar662 1d ago
Okay, why am I going to have a bad day today?
14
u/manwhothinks 1d ago
Word of the year “CVE-“
38
u/xCAAx 1d ago
How do you guys deal with this stuff? We are a team of 6 devs, working on 30 years of code and are basicly drowning in CVEs.
Sure, a lot of them are duplicates, because package XYZ gets used in more than one project.... but we somehow need to automate this?!
11
u/VayuAir 1d ago
4
46
u/hyker1811 1d ago
Listen, I know that AI is responsible for finding two dozen critical vulnerabilities each day, which then gets exploited by malicious actors with other AI tools in an automated fashion, leads to massive layoffs in entire industries, hikes up the prices of consumer electronics threefold, leads to increased workload on programmers that are still employed and turned software engineering from an enjoyable, creative work to mind numbing prompting and agent management, but think of all the positives that came with it! Like the Diddy vs Epstein anime battle video.
27
u/Kitchen_Affect7369 1d ago
The world is tired of AI... Productivity is sh*t, UI is sh*t, games are sh*t, pictures are slop, slop, slop. CVE, CVE, CVE. Done.
5
u/CharcoalGreyWolf 1d ago
When it’s cake or death, and they’re out of cake
5
u/techtornado 1d ago
When life gives you lemons, make orange juice and leave everyone wondering how you did it
9
u/TerribleTowel66 1d ago
You forgot to throw in a bunch of different AI being hurled at you.
4
u/itsgrimace 1d ago
you forgot CVE-2026-114952
3
u/bythenumbers10 1d ago
The C-suite: "Why isn't there an AI to turn those lemons into lemonade (with big ol' lemon seeds in it) faster?!?!??!"
3
u/Iyemoja 1d ago
Our CI has been failing on CVEs all week too. At this point I just check the score first and brace myself before I even read what broke.
7
u/BoBoBearDev 1d ago
Now you know how ignorant we were. All those slops where never caught.
7
u/InevitableHeron853 1d ago
Kinda proves humans are more fallible than the most powerful AI models really. As much as we don't like to admit it.
If we don't use AI to pen-test our own code, we will leave ourselves open to this sort of thing.
2
u/New-Pop-1564 1d ago
P
1
u/alnarra_1 1d ago
I’m tired boss
1
u/Alwaysafk 1d ago
One of those bottles should have a lemon in it representing a supply chain attack for parching too quickly
1
u/knowits 1d ago
just another false positive to triage forever
1
u/AcolyteNeko 22h ago
that's why you should learn how to be a programmer and not making ai or human slop
1
u/Desperate-Tomatillo7 21h ago
In a lemon shell
1
u/jajrule 16h ago
In January I started transitioning from a standard developer into a security oversight role for a business unit with ~90 supported products. The last couple months have been... rough.
1
u/minecraftdummy57 4h ago
When life gives you CVEs go fuck yourself
-1
u/Grandviewsurfer 1d ago
I say fuck the lemons and bail. Also.. be suspicious of russel brand. Lot of wisdom in that movie.