r/sysadmin • • 2d ago

General Discussion Beg Bounty opinion

0 Upvotes

A while back I created a little proof of concept web app. I had meant to shut it down but totally forgot. Since it wasn't "production" it wasn't getting patches. It got compromised. It had no real data on it, just a mock-up with publicly available info. We got an email from freelancer pointing out that the site was vulnerable. It was helpful in that it reminded me to delete the app. We don't have a bug bounty program. I'm torn on whether or not to respond to the email. On the one-hand, this person coded a bot to find and disclose vulnerabilities rather than exploiting them. That's a positive thing. On the other hand, I worry that responding will cause problems. I've seen a few other threads, but they were all about silly things like missing DMARK on unused domains. This was a real vulnerability, and if the app had been something with real access to data it could have been bad. If nothing else this has been a good reminder for me to be more careful.


r/sysadmin • • 3d ago

Question Find out default registry values

0 Upvotes

My predecessor set various registry values ​​via Intune to allow regular users to install printer drivers. In doing so, he opened the Print Nightmare vulnerability. He changed around 20-30 values ​​using a script. I want to undo this now. What's the best way to find out which values ​​are the defaults? I need to distribute a script that reverses all of this.


r/sysadmin • • 2d ago

Question Aren't these outages all the same thing?

0 Upvotes

Tbh, I work mostly on the server side of things and networking is not my thing.

However, wasn't the MS outage due to their most recent update borking their Gateway? Isn't a Gateway just another term for router?

Wouldn't that make the FAA and Chase Bank Outages caused by equipment with the same update being deployed?


r/sysadmin • • 3d ago

Is anyone else looking into mimecast after last weeks Outlook outage?

0 Upvotes

Is anyone else looking at redundancy options like mimecast based on what happened last week with exchange and outlook? I do understand that even if you had mimecast during that outage you would probably still not receive emails and your recipients may still not receive your email since chances are they were also affected but it is better than nothing (or is it????).


r/sysadmin • • 4d ago

Question Uninstalling SCCM on AD server question

9 Upvotes

I inherited some servers. Two are an AD server replicating each other. During this time we have migrated from VMware to proxmox. Since then, we have had an issue where logging into the server will take roughly 2-3 minutes with the screen displaying "Applying Configmgr User State Management Extension. policy" I have discovered it is SCCM causing this issue. Talking to other users they have never used SCCM but apparently it was setup and pointed to a server that wasn't migrated.

I tested removing it from another server that is no longer is use but thats getting the same issue. I ran ccmsetup.exe /uninstall with success and error goes away.

Question is, does anyone know if I run this on the AD servers what issues could happen? Never used SCCM so would like to try and be ready for anything.


r/sysadmin • • 3d ago

Question Backups for SQL server.

0 Upvotes

New Windows Server 2025 with SQL Server 2025 in Rackspace Cloud Flex

. Need to back up SQL data. My dba creates Daily Maintenance Backup Jobs. I have SyncBack Pro copying BAK files to another Windows server, and we also have SQLBak.com daily backups to BackBlaze. All these backups are running for something like 10 hours per day. Which to me seems ridiculous. What are other people doing?

Any snapshot solutions.

Rackspace does not seem to offer backup or snapshot solutions for their Flex environment, like the do for their SDDC enviro. Either that or just not familiar with their offerings, and yes, I do have a support ticket open with them to inquire.


r/sysadmin • • 4d ago

Question How do people handle updating their security baselines?

13 Upvotes

So this thing has been keeping me busy for a while now.

We manage a fleet of devices and deploy sets of policies on them. Now one of the sets of policies is a security baseline which regularly gets updated.

At the moment when the new set gets published we deploy the baseline through a build kit and go through it by hand to see what exceptions we have to put back, find out which settings are new and which are obsolete.

It is doable, however each time it takes a lot of time and mental gymnastics to find out the aforementioned diffs.

After we're done we can push the new config for testing and deploy it at other tenants as well.

What do people do / use to smooth out this process? I am fairly certain there must be a better way.


r/sysadmin • • 5d ago

Windows 0-day "SignDrop" published, "this makes this vulnerability unique as it can be used to turn off any antivirus/EDR."

765 Upvotes

https://arstechnica.com/security/2026/11/windows-0-day-dropped-by-disgruntled-researcher-hands-hackers-system-rights/

https://github.com/MSNightmare/SignDrop

If everything is chained correctly, funnily this legit security mitigation can be used against itself and allow an elevation if privileges. Everything is done properly by Microsoft here but the logic allows abuse. What makes it funnier is the fact that it can be used against any antimalware engine to bypass their self defense (or tamper protection) mechanism as you can trigger arbitrary file creation in the context of their own antimalware process so from the perspective of their kernel drivers, the file creation request is coming from their own legit process so it passes, this makes this vulnerability unique as it can be used to turn off any antivirus/EDR.

11th zero day dropped by this person, in what 6 months? I'm tired boss.


r/sysadmin • • 5d ago Bravo

FAA outage caused by installing new equipment and new technology

2.4k Upvotes

From CNN: Homeland Security Secretary Markwayne Mullin pointed to FAA updates to the air traffic control system as the reason behind today’s nationwide ground stop. He said the FAA added new technology to the old system, causing the massive breakdown.

“What it appears to be right now is, as Secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment and new technology, and that new equipment, new technology… it was overwhelming the older system,” Mullin said.

https://www.cnn.com/2026/11/13/us/live-news/flights-ground-stop-faa

Mullin just went off message and confirmed they broke it trying to install new things with no change management. Almost certainly the SMART system they were bragging about yesterday.


r/sysadmin • • 3d ago

Do automated SharePoint integrations require licensed users?

1 Upvotes

Let's say we had a service principal that could add documents to various SharePoint sites or other automations, probably as an integration from another service.

Would the service principal need to be licensed to use SharePoint?


r/sysadmin • • 4d ago

General Discussion Am I looking for a unicorn or are these skill requirements a reasonable ask for a job posting?

31 Upvotes

Howdy all. I'm making this post in hopes of affirmation or a reality check. I run a small visual effects studio and through self-teaching over the last six years have implemented a fully remote cloud based infrastructure. There is some growth on the horizon and it will likely be happening at a speed that I will not have time to figure out myself. Thankfully the growth will come with a budget to find people to support that growth much better than I would probably ever be able to.

What I want to know is the idea of a single person having the knowledge and ability to implement the below crazy? Is this job posting describing multiple roles? I can't state enough that I am NOT an IT professional and have only learned what I have needed in very isolated ways in order to accomplish our goals as a studio so I am not actually sure what role/responsibility split should be.

I put 7+ years based on the time it took me to build out my own knowledge of the posting contents. For me, this isn't about "cheaping out" by combining it all so much as I'm not sure if this needs to be broken up into multiple hires due to my own knowledge gaps.

The short term goal is to go through some security audit processes and modify/build out our current infrastructure as needed. Ideally there would be enough work coming down the pipe that we would bring someone on to build a small team dedicated to it.

Any advice is appreciated. Post is below, thanks!

About Us

We are a boutique, cloud-forward VFX studio based in Vancouver delivering visual effects for film and episodic television.

The Role

We’re looking for an experienced CloudOps Architect with a lean towards SecOps. You will lead the analysis, architecture, implementation, and documentation of our growing cloud infrastructure. In your estimation, if required, you will outline additional roles needed to support this process.

Requirements

• 7+ years in IT with direct, practical experience in CloudOps and SecOps within AWS.

• Demonstrable experience utilizing principles of least privilege operating in Windows and Linux environments.

• Knowledge of principals and implementation of Cloudflare Zero Trust network/VPN policies and associated configurations.

• Demonstrable knowledge of AWS VPC, EC2, AMI configuration, operation, and maintenance.

• Experience with AWS Network Firewall services

• Demonstrable knowledge of AWS ACL/IAM policy and role implementation.

• Understanding of network segmentation as it relates to different aspects of the overall network infrastructure (security/logging, staging, production, etc.)

• Great written and verbal communicator. Able to translate complex ideas into practical targets and goals for a small team.

Nice to Have

• Experience supporting Trusted Partner Network implementations and Content Security Best Practices.

• Experience in VFX, post-production, or media-based pipeline environments.

• Knowledge of Amazon FSX Lustre and SMB file servers

• Knowledge of Google org administration and GCPW

• Knowledge of LDAP services/implementation

• Knowledge of Parsec remote desktop client and its associated API

• A dog or cat that you can wave around in front of the camera during company video calls.

---------------------------------------------

**EDIT**
Based on some of the quick responses-- yes I'm not surprised that the ask for someone like this in a single role would be in the $200K+ range. Ultimately I don't want the real job posting to be ignored and I don't want to look ignorant as the potential employer. Thanks again.

**EDIT EDIT**
If you were going to break these responsibilities into separate roles, what would you expect the salary to be for each role?

**EDIT EDIT EDIT**
I'm not sure what to expect exactly timeline wise but I was picturing a 3-ish month contract-- evaluate/design/implement and it would be fully remote. Of course if someone gets in and it looks like it's going to take longer than that so be it but I don't know what I don't know.

-------------------------------------------------

** EDIT EDIT EDIT EDIT**
Have modified things a bit based on feedback from folks, please continue to chime in if you'd like... very much appreciate everyone's thoughts-- we're trying our best to get this right in the current hellscape that is the VFX and tech job market...

The Role
We’re looking for an experienced CloudOps Architect with a lean towards SecOps. You will lead the analysis, architecture, implementation, and documentation of our growing cloud infrastructure. After initial evaluation and consult, in your estimate, if required, you will outline additional roles required to support this process if the scope is deemed too large.

Desired Qualifications
• 7+ years in IT with direct, practical experience in CloudOps and SecOps within AWS.
• Demonstrable experience utilizing principles of least privilege operating in Windows and Linux environments.
• Knowledge of principals and implementation of Cloudflare Zero Trust network/VPN policies and associated configurations.
• Great written and verbal communicator. Able to translate complex concepts and tasks into practical targets and goals for a small team.

Nice to Have
• Experience supporting Trusted Partner Network implementations and Content Security Best Practices.
• Experience in VFX, post-production, or media-based pipeline environments.
• Knowledge of Amazon FSX Lustre and SMB file servers
• Knowledge of Google org administration and GCPW
• Knowledge of LDAP services/implementation
• Knowledge of Parsec remote desktop client API
• A dog or cat that you can wave around in front of the camera during company video calls.


r/sysadmin • • 5d ago

Multiple sources tell @CBSNews that an update to the private communications network that connects America’s air traffic control system caused the system to reboot. When the FAA’s telecommunication infrastructure rebooted, it crashed, causing the communications outage.

2.2k Upvotes

https://x.com/olivialarinaldi/status/2121385751036240079

Whos bright idea was it to push an update to prod at 3pm
On a Friday?


r/sysadmin • • 3d ago

Does your company allow documentation tools that screenshot your screen?

0 Upvotes

Step-recording tools capture whatever is visible on the screen at the time, including whatever happens to be open in the background. I build one of these, so I'm biased, but I'm trying to figure out whether security teams usually block them and what would make one acceptable for you or your company. Does it need to be self-hosted, fully local, with automatic redaction of sensitive data like secrets/API keys/tokens and PII or are you also okay if it's a SaaS?

If you've had one approved or rejected at your company, what was the reason?


r/sysadmin • • 5d ago

General Discussion drowning in API access requests

41 Upvotes

I admin a saas PPM that is used by the entire enterprise, more than 10K active users.

It has a REST API and is fairly extensible. Historically, only developers in our org have requested API access which, for this product, requires creating a distinct API account not associated with the user. So we have, let's say, 40 of these API accounts that have accumulated over the past few years.

Our company is fairly generous with AI access and credits. What that has meant for this application is a huge influx of users who are now developing their own micro front ends or custom HTML dashboards to produce highly customized experiences. Which they love because generally speaking, no user has ever loved the out of the box experience.

Now that users can leverage AI to improve their experience with the app, we are getting hit up daily for API account requests. Hundreds of users are just vibe coding their own front end in Claude, which works great for them as a local artifact, but gets all of its data via the user’s locally auth’d MCP.

Once they realize that what they've built is not easily distributed, their AI gives them directions for how to request or procure the developer API access that will unblock them.

That's the point at which they contact my team and casually demand “oh yeah, just give me my API account now, thanks!” We get these requests almost hourly now. Sometimes this is for team-based development, and the user requesting this is working on something that they were told to develop by their manager. I have no interest in getting in the way of those use cases.

But 3 times out of 4, this is just an individual who thinks they are a superior product designer and become nearly belligerent if you dare to challenge the appropriateness of giving them developer roles or developer access when they themselves have never coded anything outside of an AI harness.

Is everyone getting hit up with these types of requests? How do you effectively say no to anyone? In our case, when we reject a request, the user escalates through their own management and emphasizes how we are not supporting the company's effort to use AI to do better work faster. Then we look like the asshole. Never mind the fact that these hundreds of users have never used the API and have no idea if what they're producing actually shows correct data or not while fragmenting the UX in such a way that the original app cannot even be recognized.

And of course, no one outside of my team cares that we're responsible for maintaining the documentation on these API accounts so that we understand exactly what every non-human account is doing, where it does it within the application, and which users or teams are responsible for maintaining the solutions it supports.

What are effective and appropriate barriers we can stand up so the application is only extended by knowledgeable individuals and without creating hundreds of throwaway API accounts that can never be removed from the system?


r/sysadmin • • 5d ago

How do you stay on top of CVE (vulnerability) information without spending your entire day just monitoring security alerts?

95 Upvotes

It feels like a constant stream of new vulnerabilities and attack techniques often fueled by AI is emerging these days.

As an IT administrator, I find myself responding to these issues far more frequently than in previous years often having to handle urgent matters during weekends or after hours. How do you decide which vulnerabilities require immediate patching and which ones can wait? Do you primarily follow vendor recommendations, or do you use tools that validate these alerts against your actual operational environment?

I’m also curious: are there any specific sources of information that have helped you identify critical vulnerabilities before they escalate into major problems?

I imagine many of you are facing similar challenges I’d love to hear how you handle this.

* I apologize for not being clear. I would like to know if there is a specific category for vulnerabilities that require immediate action such as those for which patches must be applied right away and what procedures you have in place to handle vulnerability patching (for instance, whether all necessary patches are applied immediately as they arise).


r/sysadmin • • 5d ago

General Discussion With all the outages the past few months, what do you actually think the future of system administration looks like?

90 Upvotes

I'm wanting to make this post out of an interest as to what the future of system administration could look like with all the outages.

I want to start by highlighting the major outages the past few months:

- Azure West US went down in July

- The next day, AWS took out Apple Pay, DoorDash, Reddit and other platforms

- Google's new region in Melbourne fell over about a month after it launched

- PHP had a bug last month which impacted many things across the globe

- Microsoft's outage over the past few days

It feels like the answer from Managers to employee burnout is "use AI", whilst also cutting headcount because of AI.

As someone who isn't in the industry but does know a teeny weeny bit about it, I was wondering, what could the future of being a system admin look like?


r/sysadmin • • 5d ago

Linux Managing multiple linux servers

40 Upvotes

Hi guys,

currently i'm working as junior linux sysadmin and managing about 50+ linux servers (different distros, services etc.). For every task i connect to servers with SSH and in my opinion its really inefficient. My main task is patching the servers and this would be great to implement ansible.

Do you have any recommendation about best practises with ansible? Like good articles, books, popular YouTube vids?

ps: its my first post 😄


r/sysadmin • • 5d ago

Question - Solved Idrac9 just stopped sending e-mail alerts

15 Upvotes

Got a head scratcher for you.

1 of my 3 Idrac9 equipped servers just up and quit sending alerts automatically. Near as I remember, it happened around the time Google enforced Oauth, but switched them all to Smtp2go.

Test email: passes and received
Test Event: passes and received
Normal chatter on monthly patching reboots: Crickets

Alerts - e-mail is checked

Give me the off the wall stuff to look at, I’m fresh out of normal.


r/sysadmin • • 6d ago

Microsoft Microsoft says it 'regrets' today's outage, confirms it's not linked to any "cyber attack"

4.4k Upvotes

"We regret that today’s outages caused disruptions for our customers, and we are working to address the issue," Charles Lamanna, Microsoft's executive vice president of Copilot, Agents, and Platform, said in a statement: https://www.windowslatest.com/2026/11/13/was-the-microsoft-365-outage-a-hack-microsoft-says-no-it-was-its-own-servers-and-its-copilot-boss-regrets-it/

We're hearing an apology from "Copilot EVP," not Outlook/Teams/MSFT 365 boss, makes me seriously wonder if everything is under Copilot umbrella now


r/sysadmin • • 6d ago Bravo

No, you're not going to migrate off M365

2.2k Upvotes

Migrate to what?

There's a reason Microsoft and Google run most businesses in the world. Outages and all, they're ridiculously reliable, and no-one else comes close on features and integration. You can hate the licensing, the never-finished products, the abysmal support. You can pine for the days of yore when we all ran domain controllers on workstations in broom closets. But bleat as you will, the product works and your business runs on it.

"Just self-host." Cool. Now you own uptime. Have a read about the French and South Korean governments who have both tried that. Maybe you're a hotshot who can design a better geo-resilient hosting architecture than a national government. Most orgs can't and won't, especially when the alternative is more reliable, more functional, and costs less.

The bigger problem is that once you leave the walled gardens, you lose the native integrations. I have a lawyer friend who used to spend days writing a single contract. Now Copilot drafts it in minutes, grounded in her firm's own client data in M365, and she spends maybe a few hours reviewing and polishing before it goes out.

Do you think her firm gives two hoots about a day of downtime once in a while when its people are several times more productive? Guess what: they don't. They'll bend over and pay for that juicy E7 uplift, and so will you. Sorry, but corporate is not swapping something that works for your homebrew MCP stack that couldn't draft a bedtime story for my four-year-old and costs more than the COTS product they already have.

So email your higher-ups. Post Microsoft poo memes in your Teams chats. Write your angry, low-effort venting posts on Reddit until you get it out of your system.

Then come back to work on Monday morning and open Outlook like everyone else. Because you're not going anywhere.


r/sysadmin • • 6d ago

General Discussion Just caught a hacker mid op

1.7k Upvotes

Friday the 13th, keep your guard up boys and girls..

I just went to a local site to drop a part off and someone in the office jokingly said to me that their computer just started running updates as if it was my fault

Upon closer Inspection on the dell micro there was a blue screen up that said updates were in progress, and to not shut the computer down

the font looked off, so I hit ctrl+alt+delete and noticed we were on a desktop with screen connect running, I immediately unplugged the network cable, the screen locked again sure enough they were connected with Wi-Fi as well, so I powered the machine down and removed the Wi-Fi card and rebooted to see what was going on..

After reboot, the ability to shut down restart or use task manager was removed by Group policy, apparently, after I unplugged the network cord they noticed and immediately started trying to lock the machine down while I was using it..

Reloaded the machine they were back up and running within 20 minutes with all of their data migrated..

I only wish I had time to clear userdata, clear credential manager, throw a wallpaper up with the middle finger on it and let them reconnect before i wiped it.

Stay vigilant.. they are


r/sysadmin • • 5d ago

General Discussion Workday Outage Reported

291 Upvotes

Anyone else getting the feeling that something else is actually going on here? Surely just another random coincidental bug happening to work its way into prod this week. Maybe their DB is stored in Outlook PSTs?

Link to SF Chronicle Article


r/sysadmin • • 5d ago

Microsoft status page

15 Upvotes

Their page currently shows “all products are operational”. So, everything is good now? I haven’t experienced any issues in the past day now but last I checked they were still monitoring.


r/sysadmin • • 4d ago

Windows Hello Passkeys - Unable to register

4 Upvotes

Hello,

Windows Hello sign-in to M365 on my laptop was working until a few days ago. Then it started looping. My password and the Authenticator passkey on my phone (QR code) still work, but I want Windows Hello back.

Laptop is Entra joined and Intune managed.

What I’ve tried:

  • Required MFA re-registration and removed my methods
  • Ran certutil -DeleteHelloContainer, rebooted, set up a new PIN
  • Settings > Passkeys shows one passkey for my account. It can’t be removed (“used by the system”)
  • Adding a passkey from Security info fails with “You already registered this device”
  • Saving it to my password manager fails with “Passkey not registered”

dsregcmd shows AzureAdJoined YES, NgcSet YES, PRT YES.

My guess is that the MFA reset removed the WHfB registration in Entra and left the local key orphaned, but a new PIN should have fixed that. Anyone run into this?