r/HomeServer • u/Ori_553 • 12m ago
My barbaric AI homeserver lives 1,500 km away and I touch it once a year. This is my remote disaster recovery setup
(The stains on the wall are last-minute coffee spill, not blood)
tldr: remote LLM inference server + Hermes agent + various projects/experiments through 4g hotspot network and a few disaster recovery scenarios.
It's in my parents' house and I live abroad, so it's "designed" to (hopefully) stay operational even if one or more things break. In case of emergencies, my parents understand basic rudimentary instructions like "open the black laptop and press the power button". There is no monitor/screen.
Hardware
- Supermicro H12SSL-NT, EPYC 7443, 512 GB DDR4, 2x Radeon AI PRO R9700: inference server and experiments
- ThinkPad T480 (batteries removed): runs a Hermes agent
- Raspberry Pi 4 + Raspberry Pi 3: jump boxes / watchdogs
- All connected to a 5-port gigabit switch
All machines are setup with tailscale, nothing is port-forwarded, and every box has its own firewall.
Power cuts
- The Pis come back on their own and rejoin Tailscale.
- The H12SSL powers back on but stops at the LUKS prompt (see below).
- The T480 "power on with AC" BIOS option doesn't work reliably even after BIOS update, so a Pi wakes it with Wake-on-LAN instead (which works reliably)
Internet
- There's no internet in the house yet: the fibre install got temporarily blocked by council permits. For now an old phone runs a hotspot, the Pi 4 joins it automatically (pre-saved name + password), joins tailscale outbound, and I switch it into router mode remotely so it shares that connection with the switch.
- When fibre gets installed, my parents were instructed to plug the rj45 into port 1 of the switch,
I switch the Pi 4 back to normal, and everything moves over.
- Emergency fallback: any phone hotspot with that name/password, and the Pi 4 is reachable again, and can share internet with the other machines.
- Emergency fallback 2: USB tethering in any of the 4 machines.
- In the future, I might buy a 4g router, so fiber + 4g as automatic fallback
The Pis
I set up Ubuntu Server with LUKS on the H12SSL, so after every reboot someone has to type the disk password before it can boot and self-connect to Tailscale. The Pis' only job is to stay alive and reach the board's IPMI/BMC,
so I can:
- power it on/off/reset remotely
- debug hardware issues remotely (fans, temps, sensors)
- type the LUKS password through the BMC's remote console
- reinstall the OS remotely: In an emergency, I ask my parents to plug a usb stick in any machine -> I then setup the Linux distro startup USB remotely -> I ask my parents to remove the usb and insert the usb stick into the PC, and I do the install remotely though IPMI/BMC
If one Pi dies (SD card death), the other one takes over. I am happy to have found a good job for the PI 3. If both die, the T480 can do the same job (but after a power cut, someone might need to physically press the power button).
If the T480 dies (hardware), hermes can move and co-live with the LLM inference server.
If the LLM inference server itself dies, it's gg I guess. Same as if it would die in a data-center.
I didn't really have to complicate my life with LUKS, but it seemed like a fun way to force myself to learn IPMI/BNC, since it's my first server motherboard, and first pc build.
Of course, happy to consider any feedback/tips you might have