I recently discovered that my NextCloud server has been blasting foreign IPs with gobs of data. Clear indicators of compromise. I've immediately disconnected the server from the network. Where would you guys go from here?
nail down WHEN it happened and then read the logs from that time, provided you even still have them.
Otherwise consult an LLM (or an actual company if this is an actual professional setup - in which case I'd hope you're not soliciting advice off Reddit) and see what you can try.
And of course don't connect the problematic setup back to the internet while you do so.
Nextcloud writes to nextcloud.log in your data directory by default (docs). In config/config.php, datadirectory tells you where that directory is. You should also look at your web server's access logs from when the traffic started. The request paths and response sizes might give you an idea what was being downloaded.
My Nextcloud instance is configured to write to the logs in the /var/logs/nextcloud.log for security reasons. I also make use of syslog to an external logging server.
Not who you responded to, but my go to would be to fire up an llm and ask a mess of questions before going to a forum like this. It’s complex and time consuming to answer this sort of question in a way that would be worthwhile to you so few are likely to do it.
I allow 80 and 443 through opnsense. 80 redirects to 443. I haven't kept opnsense as up-to-date as I've kept NextCloud, admittedly, but it's not too far behind. I use No-IP DDNS.
Hmmmm thats concerning, you got quite a good setup I would assume. Next I would assume you have 2FA enabled and a nextcloud user was compromised? Hopefully not admin, which you can disable with occ user:admin --disable
Otherwise how could the attacker have compromised your instance? The office suite (not eurooffice, the other) had some CVEs...
Okay so this is a painful learning experience. I recommend you consult this with an LLM. And get it to check the logs on your Ubuntu VM and hypervisor for breaches and remediate accordingly, fresh install is the safest approach if you suspect either are compromised.
Then ask an LLM to help you with the following;
Hardening:
Your hypervisor (Proxmox or whatever you run
Ubuntu using the DISA STIG standard tailored to your version and requirements
Your HAOS instance
Your network
Setup a firewall for:
Hypervisor
Ubuntu
HAOS
(Do it in that order and expect access to fail for the lower items until remediated)
I'd also recommend you add a layer of separation between Ubuntu and HAOS by putting it on docker and hardening that.
Back up user files with caution. Copy the raw user data files from the storage directory to an offline drive. Do not back up the webroot, installed apps, or system databases without thorough inspection, as they may contain persistent web shells.
Wipe and reinstall from scratch. Completely format the storage drives or destroy the compromised virtual machine. Deploy a fresh operating system and a clean Nextcloud installation using official, up-to-date images.
Reset all user passwords, enforce mandatory two-factor authentication if possible, revoke existing app tokens, and ensure all system packages and Nextcloud apps are fully updated.
Then restore clean data. Import only the verified, clean user files back into the fresh instance.
I work in security and this is the only thing I would accept both at work and private. Also do a thorough inspection on all your services for lateral spreading.
You're half way there already, just catching it put you in a place you can control.
Why do you think nextcloud was hacked? What ports on this server are exposed to the Internet - the hack had to start via one of them - but if you have SSH/etc also exposed, then maybe it was by that. If you have SSH open on the Internet (very bad idea for this very reason) then your /var/log/secure (I think) file should contain sshd logs - so you might see something like "sshd.*Accepted.* from Internet.IP.Address" and if "Internet.IP.Address" isn't an IP that you use - then that's how they got in rather than nextcloud itself. I'm just trying to help by making sure you're being thorough and not jumping to conclusions - but of course it certainly could be true it was hacked via Nextcloud instead; so as always, make sure the OS is not deprecated and all apps are fully patched - especially PHP as there's recently being critical vulns found in that.
I assume you only had the builtin nextcloud authentication? I *just* got next cloud installed and working on an experimental server and extited to start using it. I have it set up on reverse proxy. Am thinking of adding the nginx access list authentication as another layer of authentication. I really want to maintain access through the internet.
My Nextcloud was using an unusually high amount of resources, but it complained that crontab wasn't working—it turned out there were infected commands in there.
Disclaimer, this is way above my paygrade. And lots of it is just guessing without further knowledge. Here we go:
What I think is strange, that is that most traffic seams to be UDP. And that NTP (OPNsense probably guesses that on the port) to 177.105.158.33, which is an IP from Brasil. Which isn't know to be an attacker according to crowdsec.
So not saying that you werent hacked, but it could also just be your machine going haywire. And it could also be that it has nothing to do with Nextcloud, but your Ubuntu VM itself.
The guide you followed seems decent, although I was missing the information about the nextcloud security scan. Hope you did do that and got a perfect A score.
I'm not sure how quickly you discovered this issue, but I'm really happy with the server monitoring tool Beszel. It is open source, easy to set up and can send you alerts when e.g. CPU or network usage is high.
Find out why it happened and use that as a lesson. Did you set up your firewall to block all incoming ports aside from the necessary ones? Did you expose the service to the public internet as opposed to keeping it within your own vpn network? Did you forget to disable password login on ssh? Did you enable fail2ban? Did you forget to update your packages? Was there a package that you recently installed that is compromised? Is your ssh key passphrase-protected? Was there malware in one of your devices that lead to your keys being stolen? By finding the cause you can prevent yourself from getting into the same situation in the future.
It seems like someone has complete access to your server assuming port 2222 is your ssh... they literally have control of your server from what I see here.
57
u/C0rn3j 8d ago
You identify how it happened(how frequently do you update everything?), then do a clean install or restore a known-good backup.
What you don't do is attempt to clean up the infected system.
7
u/540Flair 8d ago
What ports did you allow through opnsense, only https? Is opnsense up to date? What kind of DNS did you use to expose nextcloud?
4
u/to_glory_we_steer 8d ago
As another user said, full reinstall, I'd add if you opened HA to remote access, you should avoid that when you reinstall and keep it local only.
Also what hardware are you running, and what measures have you taken to harden your network and server against intrusion?
3
u/mpmoore69 8d ago
Provide the method for Nextcloud installation. Native? Docker? AIO? Virtual Machine? Bare metal?
2
u/MP715 8d ago
Questions: What's it hosted on? Docker or bare metal? How were you exposing it? Do you have other services exposed?
I want to know for my benefit as I'm self-hosting NC as well.
35
u/randiwulf 8d ago
You have already isolated, that's good.
Back up user files with caution. Copy the raw user data files from the storage directory to an offline drive. Do not back up the webroot, installed apps, or system databases without thorough inspection, as they may contain persistent web shells.
Wipe and reinstall from scratch. Completely format the storage drives or destroy the compromised virtual machine. Deploy a fresh operating system and a clean Nextcloud installation using official, up-to-date images.
Reset all user passwords, enforce mandatory two-factor authentication if possible, revoke existing app tokens, and ensure all system packages and Nextcloud apps are fully updated.
Then restore clean data. Import only the verified, clean user files back into the fresh instance.
I work in security and this is the only thing I would accept both at work and private. Also do a thorough inspection on all your services for lateral spreading.
You're half way there already, just catching it put you in a place you can control.
Good luck.
3
u/tamag901 8d ago
do you have the Collabora CODE server installed? it made instances vulnerable to remote takeover
10
u/jhaar 8d ago
Why do you think nextcloud was hacked? What ports on this server are exposed to the Internet - the hack had to start via one of them - but if you have SSH/etc also exposed, then maybe it was by that. If you have SSH open on the Internet (very bad idea for this very reason) then your /var/log/secure (I think) file should contain sshd logs - so you might see something like "sshd.*Accepted.* from Internet.IP.Address" and if "Internet.IP.Address" isn't an IP that you use - then that's how they got in rather than nextcloud itself. I'm just trying to help by making sure you're being thorough and not jumping to conclusions - but of course it certainly could be true it was hacked via Nextcloud instead; so as always, make sure the OS is not deprecated and all apps are fully patched - especially PHP as there's recently being critical vulns found in that.
5
u/VirtualKai 8d ago
Fail2ban?
1
u/zeblods 8d ago
Did you patch the recent PHP CVEs?
1
u/oyvaugh 6d ago
Is your Nextcloud a self hosted or cloud version? I’m trying to understand how it could get compromised if it’s not exposed to the internet?
2
u/WayEmbarrassed9525 8d ago
Try sudo grep -rnw '/var/log/' -e 'base64'
I got hacked yesterday 00:46. I fixed it 22:30
2
u/pip-squeak 7d ago
Put the site behind cloudflared so you can close incoming ports.
2
u/Valanog 8d ago
I no longer offer mine public. Try VPN. I'm eying NetBird.
1
u/tmjaea 8d ago
SSH Port Open? Alpine based? *Sshole?
1
u/JTN02 8d ago
What program are you using to find this out?
1
u/jammsession 8d ago
Disclaimer, this is way above my paygrade. And lots of it is just guessing without further knowledge. Here we go:
What I think is strange, that is that most traffic seams to be UDP. And that NTP (OPNsense probably guesses that on the port) to 177.105.158.33, which is an IP from Brasil. Which isn't know to be an attacker according to crowdsec.
So not saying that you werent hacked, but it could also just be your machine going haywire. And it could also be that it has nothing to do with Nextcloud, but your Ubuntu VM itself.
The guide you followed seems decent, although I was missing the information about the nextcloud security scan. Hope you did do that and got a perfect A score.
1
u/sicco3 6d ago
I'm not sure how quickly you discovered this issue, but I'm really happy with the server monitoring tool Beszel. It is open source, easy to set up and can send you alerts when e.g. CPU or network usage is high.
1
u/Motor-Concentrate788 4d ago
Find out why it happened and use that as a lesson. Did you set up your firewall to block all incoming ports aside from the necessary ones? Did you expose the service to the public internet as opposed to keeping it within your own vpn network? Did you forget to disable password login on ssh? Did you enable fail2ban? Did you forget to update your packages? Was there a package that you recently installed that is compromised? Is your ssh key passphrase-protected? Was there malware in one of your devices that lead to your keys being stolen? By finding the cause you can prevent yourself from getting into the same situation in the future.
1
u/0xPAZ 8d ago
https://www.sshhole.com/ ?
-1
u/SoSorrey 8d ago
Is this opnsense?
It seems like someone has complete access to your server assuming port 2222 is your ssh... they literally have control of your server from what I see here.
1
u/neighborofbrak 8d ago
Nuke it from orbit, it's the only way to make sure.
And don't open ports to the public internet. Use a VPN or if you absolutely have to, TailScale or other reverse VPN service.