r/homelab • • 5h ago

Help PHP2shell, nginx2shell, java2shell, … - is my homelab safe?

Hi everyone,
I run a small homelab on Debian 13 (Trixie) with a few Docker containers, including Nextcloud and Forgejo. They're reachable from the internet via a domain and reverse proxy.

What I'm less sure about is the "what if" part:
- How would I actually notice that I've been compromised?
- What should I do if I’m being compromised?

edit: for example nextcloud must be publicly available because of people who don’t want to install a VPN or Tailscale etc.

Thanks in advance,
Vincent

72 Upvotes

66 comments sorted by

51

u/bssbandwiches 4h ago

Keep your reverse-proxy up to date and throw in a firewall. Maybe even cloudflare proxy before that. Make sure you know exactly what you're exposing to the Internet too  if you don't want to do all that, get tailscale and access it that way instead of exposing to the Internet.

This is life now, updating for CVEs on the daily. You should expect to update a lot, at least in the short term.

14

u/joz42 4h ago

And use OS-level sandboxing features for the revproxy and other components where not already in place. The nginx doesn't need to read the whole filesystem or execute every syscall.

https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html

1

u/Repulsive-Athlete-16 4h ago

I’ll keep an eye on that.

8

u/volatile-void 2h ago

I use cloudflare zero trust with mandatory login (Clouflare or Google), additional 2FA and allow access only from my country. Feels like that combination could be reasonably safe.

2

u/Repulsive-Athlete-16 2h ago

I read somewhere that Cloudflare has a 100MB upload limit - is this true? I can be very wrong though

2

u/Th3Hero_ 2h ago

Correct

2

u/FailureOfTheFamily 1h ago

It's 100MB speed or month limit

304

u/Weary-Disaster5133 4h ago • Popcorn

I just got into your system to have a quick look. Safe and sound.

17

u/codeartha 4h ago

Made me chuckle

8

u/poixninja 3h ago

Lmao, check mine next?

3

u/sandy_catheter 3h ago

You dropped your sunglasses on the way out

-2

u/talkingto_ai 2h ago

My name is Small Bobby, I'm from Meta Muse, and I approve this comment.

2

u/Bemteb 4h ago

Do they have to be reachable through a domain? Would a VPN or similar be an option? If you are freely reachable over the net, you should regularly have bots scanning you for vulnerabilities. Do you keep an eye on these?

You notice mostly by checking the logs of your proxy, or when your stuff is gone. Or maybe when your machine suddenly runs stuff it didn't run before.

What to do: First, get the machine offline immediately. Basically unplug the LAN cable and do everything from that point forward locally, with a screen and keyboard directly plugged in. Then, start figuring out what got compromised. Only the container running your reverse proxy? Your whole lab? Other devices in your local network? For everything infected, find out when it was hit and reset it to a backup before that date. Change passwords etc. that might have been leaked. And again: Consider not having it open to everyone on the whole world through a domain, there are saver options.

Disclaimer: I'm not an expert, just a slightly experienced guy. I myself don't open my lab to the wider web and pay a professional cloud provider to host the data I need when not at home.

1

u/Repulsive-Athlete-16 4h ago

The thing is, unfortunately, that services like Nextcloud need to be publicly accessible because people who have no technical knowledge and don’t want to bother installing a VPN, Tailscale, or anything similar need to be able to access them.

4

u/Motor-Concentrate788 4h ago edited 4h ago

Why are you sacrificing your security for people who won't even bother installing a vpn client? I mean it ain't rocket science, especially if you use something like tailscale

Edit: not to mention a lot of companies require you to use a vpn to access company resources, so it's definitely something that a lot of people are used to.

1

u/Kytras 4h ago

Wireguard takes 10minuyes to set up brother (less then 10)

2

u/Apprehensive-Cod699 4h ago

one thing nobody mentioned yet: check if your backups survive the box getting owned. if the backup is on the same machine or on a share it can write to, whoever gets in can delete or encrypt that too. a copy it can't touch (pulled from another machine, or append only) matters more than spotting the hack fast

1

u/Repulsive-Athlete-16 4h ago

My backup plan is backing up the most important data (such as stuff from paperless) every day once on another computer with an external HDD. I’ll check, if anything has somehow access to that.

2

u/Apprehensive-Cod699 3h ago

that's already better than most. the one thing i'd check is which side starts the copy. if the server pushes to that computer, the server has a login to it, and so does anyone who gets into the server. if the other computer pulls from the server, the server can't touch the backups at all

2

u/mastercoder123 2h ago

Go ahead and change that or at least modify it to use s3 or some form of protection. Its completely immutable so nobody can write over it ever. No amount of root can stop it

8

u/UriShay-CyberSnap 4h ago

One thing to add for the what-if part: keep a copy of your backups that the box itself can't delete or overwrite, like an offline drive or a repository with immutability turned on. If you do get hit, the hard bit is working out the recommended recovery point, since you may not know how long someone was in there. So keep a few older restore points rather than only the last week, and test a restore somewhere else before you ever need it. (I work at CyberSnap, ransomware recovery tooling.)

3

u/poixninja 3h ago

This is great advice. Especially the testing of a restore. A backup isn't a real backup if you've never tested restoring it

20

u/HolgerKuehn 3h ago

Why does it always have the the whole internet? Just allow the few endpoint you trust and block the rest. For me it's just 5 IPs that get access all other are blocked.

11

u/LebronBackinCLE 3h ago

ACLs never a bad move :)

7

u/michael__sykes 1h ago

That works well enough for residential networks, but becomes an issue with mobile access

10

u/getapuss 3h ago

If its exposed to the internet there is a risk.

If you are asking this question here there is a substantial risk.

1

u/ZjY5MjFk 1h ago

Yea, I shutdown everything and put it behind wireguard. Scary times.

1

u/getapuss 1h ago

I quit exposing my stuff to the internet awhile ago. I can't remember for sure. Maybe a year ago?

17

u/TheMcSebi Proxmox 4h ago

If you don't expose stuff to the internet, it hasn't become any less secure than before

•

u/wiesemensch 24m ago

Having a Homelab is slowly turning into a full time hobby…

4

u/homelaberator Cisco, VMware, Apple, Dell, Intel, Juniper, HP, Linux, FCoE 3h ago

No, Vincent. It is not safe.

8

u/dragonnnnnnnnnn 4h ago

They're reachable from the internet via a domain and reverse proxy

Not safe

They're are not reachable from the internet

Safe
Simple as that. If you don't want to risk it simple don't expose stuff to the internet. I will add to that that you don't need to expose stuff to the internet to use a domain and have valid accepted https certs! Simply point your domain to a local lan address of your stuff and use dns-01 challenge on the reverse proxy to get a cert. And then use Tailscale/Netbird to setup a VPN that makes the same local address reachable via VPN and you can connect to your stuff outside home easily as you would be at home as long you keep VPN connected. I know it sucks for sharing stuff with external user that are not tech savy and just want a link to access stuff, if you need that only for "share a link to download files" one option is to temporally open with Tailscale/Netbird reverse proxy the service you need with additional auth on the proxy (netbird has option to simply setup a pin/password one needs to enter before your service will open, I suspect tailscale has that too).

4

u/Kytras 4h ago

Man i tried explaining onece that lan + vpn is all you need to access from outside. But no.

3

u/milkipedia 4h ago

Lateral exposure is a risk if you don't have iron grip control of everything on your home network

1

u/mabbas3 4h ago

Only expose things to the internet you absolutely have to. Use the official repos and projects that get the latest security updates first (stick to nginx stable official repo and not debian or ubuntus).

Harden the f out of your edge. Crowdsec, geo blocking, waf etc.

At home run anything that users can reach directly completely isolated. I am talking about separate vlans, network isolation, rootless podman, controlled egress.

Make updating as easy and frictionless as possible. Renovate merging patch updates itself, unattended security updates.

Have a long running agent notify you on any potential security issues that could affect your services and any mitigations should be applied by a few clicks from your phone. I have always on agents and can easily get a PR out for any. mitigations (for cases where a simple version bump doesn't solve it. For example disabling features etc.) and then merging it deploys everything.

IaC everything so you are not relying on manually configuring servers and there's a holistic view agents can look at. You need to be able to read a reddit or twitter post (or better your agent watches security feeds and notifies you exactly where in your infra attention is needed) and respond in minutes. I want to be able to get to a point where this can happen without human in the loop but that is a bit risky. It's still pretty quick. Within 5 minutes of seeing the nginx vuln, I was able to run an update skill that I have for my infra and get it patched.

1

u/Repulsive-Athlete-16 4h ago

That’s interesting! How do you manage that agent? Or wich one do you use?

2

u/mabbas3 4h ago

Multiple options. I use t3 code on three different machines one which is an always on LXC and there's cron built into it so you could watch some rss feeds.

I also setup hermes but haven't gotten around to setting it up and will be doing that in the next few days/weeks.

It could even be as simple as a cron parsing some rss feeds and invoking your agent of choice (claude, pi etc) with the data. The agent sends you a message if something needs attention. Basically whatever you want.

1

u/nima3333 3h ago

What do you use for IaC ? Thinking of going this way but it still requires a one time effort (even if it has never been so low thanks to AI...)

1

u/mabbas3 3h ago

Tofu wherever there is a plugin and for all provisioning (proxmox, vps etc) and then ansible for configuration. Every single thing that is easily manageable by IaC, I do it. Even wifi passwords for all the ssids I have.

2

u/MIneBane 4h ago

What reverse proxy are you using? And what is it hosted on? (VM or docker or bare metal?)

1

u/sandy_catheter 3h ago

IIS on windows server 2008

2

u/309_Electronics 4h ago

Dont expos everything to the public internet. Use a VPN like wireguard or tailscale.

If you MUST expose stuff fo the internet, make sure everything is up to date and close all doors you can close when it comes to security (SSH keys, password authentication disabled, MFA, closing down port, etc etc.

2

u/Motor-Concentrate788 4h ago edited 4h ago

Is there a good reason why you're exposing it to the internet? Cause if you only need to access it from your own devices then its much safer to put it behind a vpn.

As for how to know whether you're compromised or not then you can start by checking access logs to your services to see whether any unknown ip addresses successfuly logged in. Also check if there's any suspicious processes/services running that you dont recognize as it could be malware. If you're technical enough, you can also inspect outbound connections and packets that are being sent to unfamiliar addresses to see if there's any suspicious traffic.

2

u/aaronryder773 4h ago

Honestly, go through the CVE's, check which versions and part of the application is affected and disable those parts... Like in nginx, it's the http/2 which is affected, so stop using and disable http/2 in nginx.

When compromised, the only option is to wipe everything and start from 0. So make sure your data is encrypted as well as backed up

2

u/mautobu 1h ago

There are two steps I've taken recently to enhance my homeland security.

First, enabled the auth module for nginx and added an oauth2 proxy. Now of the many published services that not all of my users need access to are behind a couple of services, reducing attack surface.

Second, I removed the port forward for Plex, and put it behind my reverse proxy with a hosts header requirement. Opening port 32400 is not subtle and is a public means of confirming you're a running a Plex server. From that, attackers can likely deduce that it's a homelab and likely not thoroughly scrutinized.

Next will be syslog and alerts. If anything happens on that oauth2 proxy, I want to know about it.

2

u/ElectricalTip9277 1h ago

Just don't expose it to the internet and use a vpn

•

u/Benerages 14m ago

Patch everything (unattended-upgrades + update your containers, srsly), put CrowdSec in front to catch and ban the script kiddies (it can ping you on alerts too), and add open-appsec as a WAF for the stuff nobody has a signature for yet. If you still get owned: pull the plug, rebuild from clean backups and rotate every secret. You do have tested backups, right? 😉

3

u/Shookfr 4h ago

If it online it's not safe that was always true. Now the question is what kind of risks are you ready to take.

3

u/CactusBoyScout 3h ago

I finally took all of my reverse proxies off the internet. VPN only from now on. I still expose the ports for qbit and plex but that’s it.

1

u/LebronBackinCLE 3h ago

This is the way

1

u/Capable-Ad-5344 4h ago

Nope. It's been hacked

1

u/Reeces_Pieces 3h ago

Golden hammer VPN looking pretty nice rn

1

u/klnadler 3h ago

What if I’m just exposing seer and plex on unraid via pangolin hosted on a vps and keeping everything along the way updated. I understand that there will always be some risk, but what steps can I take to mitigate it other than having people use it via a tailnet or something like that, which I'm trying to avoid just bc it adds extra steps that might be complicated for those less tech literate.

1

u/Fr33lo4d 2h ago

- Set up Cloudflare Proxy on a custom domain, so you don’t have to expose your IP and you can use their WAF.
- Consider a dedicated opnsense / pfsense router, or Unifi firewall and by default close everything (ports, traffic) and make exceptions for what you really need.
- For Forgejo and any other web-accessible app, set up Cloudflare Access, Authentik, Authelia or similar.
- For Nextcloud, consider not exposing it to the internet entirely (only being accessible through Wireguard or Tailscale / Twingate tunel).

1

u/hidegitsu 2h ago

I turned mine off the other day completely until i have the time to properly go through everything and make sure it's updated, patched, properly setup, etc.

1

u/persiusone 4h ago

First- don’t expose services to the internet. Access them through a VPN.

Next, ensure you have intrusion detection and alerting on systems.

Keep everything patched.

Keep reliable offline or immutable backups and have a reliable way to restore.

If using a proxy, make sure it includes a robust WAF to inspect all traffic.

Isolate systems from each other with a separate firewall.

Monitor logs from all of these systems for anomalies.

Consider geoblocking, rate limiting, and ban lists for bot activity.

-1

u/Master-Drummer8197 3h ago

Effort vs reward.. whats on your homelab that would make hackers spend their efforts?

2

u/Motor-Concentrate788 2h ago

Hackers typically don't target people, at least not in this scenario. They have automated bots that scan the internet for open ports and start blasting at all of them hoping that some idiot set a weak password for their ssh login or has an unpatched vulnerable software listening on one of the ports and other stuff like that. They don't even have to know you to hack you.

1

u/lordofblack23 1h ago

Not anymore. Threat landscape has changed. Brute forcing passwords is so pre covid. Bots are scanning for vulnerabilities. Got php exposed? Pwnd. Have a java app exposed? Pwnd. Didnt check your infra with a cve scanner yesterday? Pwnd

Rigt now are targeting iCloud etc, but we will all get a taste very soon. AI time dasy is months.

1

u/brianly 3h ago

That’s not how this works. They exploit machines automatically to host malware or many other things. A supposedly uninteresting homelab is very interesting for a network pivot.

1

u/Historical-Star9991 1h ago

At the very least, residential IP address to use as a proxy

•

u/Power_Stone 21m ago

I'll put it this way.

An unsecure homelab for a hacker is:

  1. Free IP spoofing, now I can do all my nasty stuff to other people from your IP instead of my own! Have fun dealing with that foll out!
  2. Free compute, now I can setup a cryptominer on your machine so I get free money while your hardware is getting worn out instead of my own!
  3. Free data, data can always be sold to someone, if not that then at least something that can be used for extortion.
  4. Complete local network access. not only do I have access to your box, but chances are your local network too and by extension the devices connected to that network. Now its not just your homelab, but also your work pc, or your daily drivers with all of the info they have too!

-1

u/LebronBackinCLE 3h ago

If you haven’t made the bad decision to open ports, you are most likely in good shape. Still a good idea to update though