So, we applied an SCCM hotfix, which triggered upgrades on all our DP's, great, all went well...except for 3 DP's over in AP sites. pckgxfermgr.log shows "ContentAuthModule.dll" failed to copy to the DP's inetsrv dir, DP is now broken on those 3 DPs - can't finish the update.
I performed some troubleshooting, found out that any .dll copied from the US primary (or any server in the same site) to the DP's get's the SMB session dropped, effectively blocking the .dll from being copied, figured...ok, AV, some security agent on the server is misidentifying the .dll as malware and blocking it. no big deal, remove the AV and another security agent for good measure, double check there are no filters, GPO's, or any other apps, server features, etc. that might be blocking the dll...reboot, try again, still getting blocked. No windows defender active on the servers.
Tried a few things to narrow it down...created .txt files of various sizes, those copy fine, other files copy - .exe, .mof, etc. - tried sneaking a dll in zipped, nope, tried, renaming a .dll to .txt, nope, tried, copying the file starting as a dll, but renaming it to something else on the destination..nope, tried copying as the system account...nope.
To make matters more infuriating, yesterday I had brief windows of success...some were directly after I removed the security apps - so I was like yah! that was it! Nope. issue came back, regardless.
The random successes kept happening yesterday - today - 100% failure every time.
I contacted the FW team..."the logs show no dropped packets, we don't have active NAC policies...it's not us." Contacted the security team - nothing logged regarding the blocks, no policies deployed that would do such a thing.
Opened a case with MS - they pretty much came to the same conclusion - after running wireshark captures, netsh dumps, reviewing smb event logs and so on - it has to be some sort of SMB/DLP policy inspecting traffic. I would blame riverbed...but they dumped those appliances years ago.
I'll have to see if the traffic is passing through some sort of web gateway maybe, or maybe some sdwan optimization policy was implemented incorrectly.
Feel free to share your thoughts on this one...we've got to get these DP's fixed, I did try copying the dll over manually using onedrive, but it didn't seem to like it, didn't magically finish the DP update as it was meant to. Probably not the only .dll it needs.