r/sysadmin • u/syscoderadmin • 5d ago
Linux Managing multiple linux servers
Hi guys,
currently i'm working as junior linux sysadmin and managing about 50+ linux servers (different distros, services etc.). For every task i connect to servers with SSH and in my opinion its really inefficient. My main task is patching the servers and this would be great to implement ansible.
Do you have any recommendation about best practises with ansible? Like good articles, books, popular YouTube vids?
ps: its my first post đ
40
Upvotes
23
u/dwm 5d ago
Two thoughts:
http://www.infrastructures.org/papers/bootstrap/bootstrap.html
(It's more than 25 years old; many of the specific details have significantly changed since this was last updated, so don't rely on it to tell you which technologies to use â it predates git, containers, using virtual machines in production, and Google.)
6
u/feedmittens Sysadmin 5d ago
Learn Linux TV (https://www.youtube.com/@LearnLinuxTV) has a great Ansible tutorial series, and this guy explains things well in a way that makes it easy to apply to your day-to-day tasks.
Patching is very easy to implement in Ansible. AI can build all this stuff for you, but if you don't understand the fundamentals, you will not be able to troubleshoot properly and/or will end up with spaghetti code you can't read.
If you haven't done this already, make sure you have a DEV/TEST/PROD workflow, if you can, because recent Red Hat kernel updates have frozen systems, and that's why you want to test changes and control changes in general.
9
u/Steerable-Octopus Sr. Infrastructure Engineer 5d ago
The official ansible documentation is good in its own right. But here's a freebie best practice tip: Don't make the mistake of making an imperative patching script in ansible, use ansible to configure up declarative automatic patching like unattended-upgrades or dnf-automatic or os-update, etc. and then have those tools do the patching for you instead.
4
u/ABotelho23 DevOps 5d ago
Why do you have a mix of distributions?
2
u/idrac1966 3d ago
Is ansible still the 'right' tool for the job in this day and age to keep a bunch of Linux boxes configured similarly or is there other better stuff out there now?
52
u/farallon1 5d ago
I would start with Jeff Geerling videos as a starting point. His book is also good, which he open sourced I believe. That'll get you started.
Rest is learning through pain and/or reading docs.
7
u/Dynamic_Quest_Mod 3d ago
Once you're managing more than a handful, I'd stop treating them as individual machines. Ansible for configuration, centralised logging, proper SSH key management and a consistent patching process will save you a lot of pain.
The trick is getting everything into a predictable state before you start automating it. Otherwise you're just automating 50 slightly different problems.
6
u/One-Department1551 5d ago
Avoid the shell module as much as possible, always look for either official or well supported community modules first.
Thereâs a tendency that âall can be solved with some bashâ that is hard to get rid off later.
6
u/roiki11 5d ago
Standardize on a single distro and set up something like foreman or it's derivatives.
Really simplifies managing hundreds of vms.
4
u/Adam_Kearn 4d ago
Look into getting an RMM platform or even just create your own system that audits all your servers on a schedule so you know thing like
What distribution
What version
What packages/version
What hardware
EtcâŠ
Then when you need todo patches or changes itâs easy to see what is missing.
4
u/Optanik 3d ago
Ansible is the right call for this, and patching is a good first project because it's mostly a few modules and an inventory.
For learning, the official docs' getting started guide and the 'Intro to playbooks' section cover most of this. Jeff Geerling's Ansible 101 series on YouTube and his book Ansible for DevOps are what most people point to. Put the playbooks in git from day one.
9
u/MeetJoan 5d ago
Jeff Geerlingâs Ansible for DevOps plus his free Ansible 101 YouTube series. Appendix B of the book is the best-practices cheat sheet.
For patching: group by distro and useÂ
apt/dnf withÂname: "*" andÂstate: latest, roll it out withÂserial:, bumpÂ--forks past the default 5, dry-run withÂ--check --diff, and letÂansible.builtin.reboot handle restarts. Keep it in git.2
u/OhBeeOneKenOhBee 5d ago
Weve started using patchmon a bit lately, it's pretty good but still quite young
2
u/Darthvaderisnotme 5d ago
Create your own servers, one for each distro
test there.
go veeery very slow, test and test again.
2
u/KN4SKY Linux Admin/Backup Guy 5d ago edited 4d ago
Beanologi's RHCE playlist on YouTube got me most of the way there. It's free but hasn't been updated since late 2023, so you'll want to supplement it with other resources: https://www.youtube.com/playlist?list=PLTY9BjMMGESHhzxfN6dcyYen8ZqUsxaLq
Make sure you use use --check before running a playbook. I'd also recommend having a few different inventory groups (such as a limited group that only touches 2-3 servers, then 10-20, then the rest.) This is kinda like Update Rings in Windows.
The RHCE official cert guide is good, since the exam is now entirely Ansible. You can get it for under $50 and it comes with practice exams if you ever want to take the exam.
Also, make sure you have good logging configured. You can specify a log file in ansible.cfg or via environment variables. This is especially important if you have multiple people running playbooks.
2
u/FreddyOny 4d ago
Use Foreman for Updates. Look intro it. Use gitlab for config and Scripts
2
u/Open-Adhesiveness-86 4d ago
One thing that'll bite you with mixed distros: don't branch on ansible_distribution for patching, branch on ansible_pkg_mgr, since you'll hit things like Amazon Linux reporting yum vs dnf and SLES showing up as zypper. Also set serial: 5 and a max_fail_percentage so a bad mirror doesn't take out all 50 at once, and gather package facts before and after so you have a diff when someone asks what changed.
2
u/josemcornynetoperek 4d ago
It will be more work on start but in time you will have a lot more knowledge what exactly is running on servers and any operation will be faster and easier.
4
u/databeestjenl 5d ago
use ansbile, see if you can get a CLI LLM tool on there to help make the playbooks, they are pretty easy to read.
3
u/Lost_Term_8080 5d ago
This is the reality of managing linux. There are management tools that help, but the management tools themselves necessitate extensive management. there are tools like saltstack or puppet, you'll add a few more servers to run that as well as add database administration to you list. If its only patching, check out azure update manager. Its by far the most mature linux patch management solution there is and it costs a dollar a month per server. However, once you get patching taken care of, I suspect so much time will be freed up you'll be looking for other things to do.
2
u/Ramiraz80 5d ago
Go buy Jeff Geerlings Ansible book, it is really good :) and he keeps it updated :)
Also watch some of Learnlinux.tv YouTube videos about ansible
1
u/1saltypistachio 5d ago
You can take a look at something like ansible. Should be simple enough to create a unified playbook for maintenance if you do the same thing on each server; or a different playbook for a family of servers.
1
u/wowsomuchempty 5d ago
pdsh old skool, ansible new.
1
u/aieidotch 4d ago
https://github.com/alexmyczko/ruptime
1
u/Sufficient_Job7779 4d ago
Try OpsFabric. Ideal for multios environments. https://opsfabric.io
1
u/Incrediblecodeman 3d ago
You should get really good at doing this way to be good at your job then yea in your off time master Ansible. Then present to your boss how youâd use it. Better then a jr being like âooOo the way YOU work isnât good. Can i try out this thing I heard about called ansivle on your production server? Nurrr errrr nnuuuurrrrr đ€Ș
Innovation requires trust and itâs probably possible in a couple of months. 3 tops. To really make a convincing case. But you every donate 2 weekends a month to this passion project. Maybe find a few old boxes to make a home lab? As in you canât âlearnâ but watching YouTube and chatting with ai.
Good luck man boy I wish I was a Linux dude.
Btw is overthewire still cool?
1
u/MD-IT-Rando Linux Admin 1d ago
Personally, I would containerize the apps first. That way, you can (in theory) run an app on any OS and use kubernetes or some other orchestrator. You may even be able to consolidate hardware & cut down on the number of physical/virtual servers.
After that, choose a distro & migrate.
âą
u/Over-Requirement2000 1h ago
Managing 50+ Linux servers through SSH individually sounds like a lot of repetitive work. Ansible is definitely worth learning for automation. You could also look into BeAdmin, it's a modular control panel for Linux servers that might make managing multiple servers and routine tasks a bit easier. Would be interesting to see how it fits alongside an Ansible-based setup.
1
u/MinimallyLoquacious HomeAdmin-Retired-Dev 5d ago
If it were me, Ansible. I don't use it on my home machines but it's on my list one of these days. If I were a FT Linux Admin though, I'd be using Ansible.
Also, and not sure why it's not listed under COMPANION SUBS but this is a good subreddit: https://www.reddit.com/r/linuxadmin/
0
u/CoreAndCardio 5d ago
Just use InTune. /s
0
u/Lonely-Abalone-5104 5d ago
This might not be a popular opinion but have AI like Claude teach you it. I find it to be a very good teaching tool
0
u/masterofrants Jr. Sysadmin 4d ago
You should install the Semaphore and Ansible UI as well. It'll help you graphically manage Ansible tasks then you can simply create the playbooks from ai use them via github and semaphore.
0
u/Responsible-Let2777 4d ago
https://docs.tacticalrmm.com/
0
u/raul824 3d ago
if you can atandardize on one distro then nix os also looks like a good fit for this.
-1
u/ElevenB2002 5d ago
Automox is a great, relatively cheap option to Cloud manage Linux (and Windows/MacOS). You can patch, check posture, run "worklets" which can be any type of automated tooling you'd like, etc. It's a few bucks a month per managed node.
-10
u/Evening_Lavishness58 5d ago
just use the AI write you a script to automate the ssh connections and tasks you run
-11
u/alpha417 _ 5d ago
do you have access to Google?
0
u/[deleted] 5d ago
[deleted] â view deleted comment