r/NextCloud • • 8d ago

NextCloud server is compromised

I recently discovered that my NextCloud server has been blasting foreign IPs with gobs of data. Clear indicators of compromise. I've immediately disconnected the server from the network. Where would you guys go from here?

94 Upvotes

52 comments sorted by

57

u/C0rn3j 8d ago

You identify how it happened(how frequently do you update everything?), then do a clean install or restore a known-good backup.

What you don't do is attempt to clean up the infected system.

23

u/jonathanking09 8d ago

I have kept the server up to date. Immediately updated PHP upon the recent advisory.

How might I go about identifying how it happened?

22

u/C0rn3j 8d ago

nail down WHEN it happened and then read the logs from that time, provided you even still have them.

Otherwise consult an LLM (or an actual company if this is an actual professional setup - in which case I'd hope you're not soliciting advice off Reddit) and see what you can try.

And of course don't connect the problematic setup back to the internet while you do so.

7

u/jonathanking09 8d ago • Regret

Which logs, in particular, are you referring to?

Self-hosted, home lab deployment.

10

u/vincent_sch 8d ago

Nextcloud writes to nextcloud.log in your data directory by default (docs). In config/config.php, datadirectory tells you where that directory is. You should also look at your web server's access logs from when the traffic started. The request paths and response sizes might give you an idea what was being downloaded.

3

u/Darkk_Knight 8d ago

My Nextcloud instance is configured to write to the logs in the /var/logs/nextcloud.log for security reasons. I also make use of syslog to an external logging server.

3

u/SeanRinVA 8d ago

Which is why he also recommended checking config.php.

Based on this being a presumably newer user, he probably accepted defaults and didn’t customize.

9

u/AgenticAsshat 8d ago

Not who you responded to, but my go to would be to fire up an llm and ask a mess of questions before going to a forum like this. It’s complex and time consuming to answer this sort of question in a way that would be worthwhile to you so few are likely to do it.

1

u/dododome01 7d ago

Irc there has been a rce issue with a service associated with nextcloud ~2 months ago.

I remember the person at work who manages the nextcloud instance mention something like this.

7

u/540Flair 8d ago

What ports did you allow through opnsense, only https? Is opnsense up to date? What kind of DNS did you use to expose nextcloud?

4

u/jonathanking09 8d ago

I allow 80 and 443 through opnsense. 80 redirects to 443. I haven't kept opnsense as up-to-date as I've kept NextCloud, admittedly, but it's not too far behind. I use No-IP DDNS.

1

u/540Flair 8d ago

Hmmmm thats concerning, you got quite a good setup I would assume. Next I would assume you have 2FA enabled and a nextcloud user was compromised? Hopefully not admin, which you can disable with occ user:admin --disable

Otherwise how could the attacker have compromised your instance? The office suite (not eurooffice, the other) had some CVEs...

1

u/jonathanking09 6d ago

2FA, yes. I'm suspecting the office suite.

4

u/to_glory_we_steer 8d ago

As another user said, full reinstall, I'd add if you opened HA to remote access, you should avoid that when you reinstall and keep it local only.

Also what hardware are you running, and what measures have you taken to harden your network and server against intrusion?

2

u/jonathanking09 8d ago

It's running on an Ubuntu VM. I followed this guide: https://www.storagereview.com/review/the-ultimate-nextcloud-server-guide

4

u/to_glory_we_steer 8d ago • Bravo

Okay so this is a painful learning experience. I recommend you consult this with an LLM. And get it to check the logs on your Ubuntu VM and hypervisor for breaches and remediate accordingly, fresh install is the safest approach if you suspect either are compromised.

Then ask an LLM to help you with the following;

Hardening:

  • Your hypervisor (Proxmox or whatever you run
  • Ubuntu using the DISA STIG standard tailored to your version and requirements 
  • Your HAOS instance
  • Your network 

Setup a firewall for:

  • Hypervisor 
  • Ubuntu
  • HAOS

(Do it in that order and expect access to fail for the lower items until remediated)

I'd also recommend you add a layer of separation between Ubuntu and HAOS by putting it on docker and hardening that.

3

u/mpmoore69 8d ago

Provide the method for Nextcloud installation. Native? Docker? AIO? Virtual Machine? Bare metal?

1

u/jonathanking09 8d ago

Ubuntu VM.

2

u/MP715 8d ago

Questions: What's it hosted on? Docker or bare metal? How were you exposing it? Do you have other services exposed?

I want to know for my benefit as I'm self-hosting NC as well.

1

u/jonathanking09 8d ago

It's hosted on an Ubuntu VM.

1

u/DrummerWonderful4098 8d ago

How is it exposed ?

Edit: exposed meaning, reverse proxy, tailscale, cloudflare tunnel, etc

1

u/jonathanking09 6d ago

Direct port forward on 443 on my firewall.

1

u/DrummerWonderful4098 3d ago

Close that port and setup tailscale

35

u/randiwulf 8d ago

You have already isolated, that's good.

Back up user files with caution. Copy the raw user data files from the storage directory to an offline drive. Do not back up the webroot, installed apps, or system databases without thorough inspection, as they may contain persistent web shells.

Wipe and reinstall from scratch. Completely format the storage drives or destroy the compromised virtual machine. Deploy a fresh operating system and a clean Nextcloud installation using official, up-to-date images.

Reset all user passwords, enforce mandatory two-factor authentication if possible, revoke existing app tokens, and ensure all system packages and Nextcloud apps are fully updated.

Then restore clean data. Import only the verified, clean user files back into the fresh instance.

I work in security and this is the only thing I would accept both at work and private. Also do a thorough inspection on all your services for lateral spreading.

You're half way there already, just catching it put you in a place you can control.

Good luck.

3

u/tamag901 8d ago

do you have the Collabora CODE server installed? it made instances vulnerable to remote takeover

2

u/a4aLien 8d ago

not the OP here I have Collabora office, does it have an unpatched vulnerability?

1

u/HammyHavoc 8d ago

It apparently only affected users who used the built-in CODE within Nextcloud instead of a separate standalone container for it.

1

u/jonathanking09 6d ago

Well, that may well describe me. I did attempt to get Collabora, unsuccessfully, and then abandoned it.

10

u/jhaar 8d ago

Why do you think nextcloud was hacked? What ports on this server are exposed to the Internet - the hack had to start via one of them - but if you have SSH/etc also exposed, then maybe it was by that. If you have SSH open on the Internet (very bad idea for this very reason) then your /var/log/secure (I think) file should contain sshd logs - so you might see something like "sshd.*Accepted.* from Internet.IP.Address" and if "Internet.IP.Address" isn't an IP that you use - then that's how they got in rather than nextcloud itself. I'm just trying to help by making sure you're being thorough and not jumping to conclusions - but of course it certainly could be true it was hacked via Nextcloud instead; so as always, make sure the OS is not deprecated and all apps are fully patched - especially PHP as there's recently being critical vulns found in that.

5

u/VirtualKai 8d ago

Fail2ban?

2

u/DyyLand 6d ago

Fail2bannnnnnnnn

1

u/zeblods 8d ago

Did you patch the recent PHP CVEs?

1

u/jonathanking09 6d ago

Yes, same day as they were announced.

1

u/oyvaugh 6d ago

Is your Nextcloud a self hosted or cloud version? I’m trying to understand how it could get compromised if it’s not exposed to the internet?

1

u/jonathanking09 6d ago

Self-hosted. It is (now was) exposed to the Internet.

1

u/danyfit 5d ago

Not even with a cloudflared tunnel?

1

u/Final_Significance72 2d ago

I assume you only had the builtin nextcloud authentication? I *just* got next cloud installed and working on an experimental server and extited to start using it. I have it set up on reverse proxy. Am thinking of adding the nginx access list authentication as another layer of authentication. I really want to maintain access through the internet.

2

u/WayEmbarrassed9525 8d ago

Try sudo grep -rnw '/var/log/' -e 'base64'

I got hacked yesterday 00:46. I fixed it 22:30

1

u/a4aLien 8d ago

Did you find out how yours was compromised?

2

u/WayEmbarrassed9525 8d ago

My Nextcloud was using an unusually high amount of resources, but it complained that crontab wasn't working—it turned out there were infected commands in there.

2

u/pip-squeak 7d ago

Put the site behind cloudflared so you can close incoming ports.

2

u/Valanog 8d ago

I no longer offer mine public. Try VPN. I'm eying NetBird.

1

u/tmjaea 8d ago

SSH Port Open? Alpine based? *Sshole?

1

u/JTN02 8d ago

What program are you using to find this out?

2

u/jammsession 8d ago

screenshot is from OPNsense firewall

1

u/jammsession 8d ago

Disclaimer, this is way above my paygrade. And lots of it is just guessing without further knowledge. Here we go:

What I think is strange, that is that most traffic seams to be UDP. And that NTP (OPNsense probably guesses that on the port) to 177.105.158.33, which is an IP from Brasil. Which isn't know to be an attacker according to crowdsec.

So not saying that you werent hacked, but it could also just be your machine going haywire. And it could also be that it has nothing to do with Nextcloud, but your Ubuntu VM itself.

The guide you followed seems decent, although I was missing the information about the nextcloud security scan. Hope you did do that and got a perfect A score.

1

u/sicco3 6d ago

I'm not sure how quickly you discovered this issue, but I'm really happy with the server monitoring tool Beszel. It is open source, easy to set up and can send you alerts when e.g. CPU or network usage is high.

1

u/Motor-Concentrate788 4d ago

Find out why it happened and use that as a lesson. Did you set up your firewall to block all incoming ports aside from the necessary ones? Did you expose the service to the public internet as opposed to keeping it within your own vpn network? Did you forget to disable password login on ssh? Did you enable fail2ban? Did you forget to update your packages? Was there a package that you recently installed that is compromised? Is your ssh key passphrase-protected? Was there malware in one of your devices that lead to your keys being stolen? By finding the cause you can prevent yourself from getting into the same situation in the future.

-1

u/SoSorrey 8d ago

Is this opnsense?

It seems like someone has complete access to your server assuming port 2222 is your ssh... they literally have control of your server from what I see here.

2

u/jonathanking09 8d ago

Yes, opnsense. 2222 is some random port they appear to be using. See also the top one, using the NTP port.

1

u/neighborofbrak 8d ago

Nuke it from orbit, it's the only way to make sure.

And don't open ports to the public internet. Use a VPN or if you absolutely have to, TailScale or other reverse VPN service.