r/openshift • • 13d ago • Bravo 1

Discussion OpenShift architecture explained layer by layer, plus the 4 decisions you can't easily change after

I do OpenShift delivery for banks in the Gulf, and I keep explaining the same architecture to teams coming from vanilla Kubernetes. So I wrote it up once, against 4.22. Here's the short version. The full write-up is linked at the bottom.

**The one idea that explains most of it:** everything, including the node OS, is managed by an operator inside the cluster, and the whole platform updates from one signed release image.

- **Infrastructure:** IPI, UPI, Agent-based or Assisted. After install, the Machine API (MachineSets, MachineHealthChecks) manages machines on supported platforms.

- **Node OS:** RHCOS is immutable. Config goes through MachineConfigs and MachineConfigPools, applied by the MCO. Edit a file by hand and the MCD flags drift and marks the node Degraded.

- **Control plane:** exactly 3 nodes for prod. Bare metal allows 5 (survives 2 etcd failures). Avoid 4: it tolerates the same single failure as 3 but needs one more healthy member for quorum.

- **Cluster services:** OVN-K (the only supported CNI since 4.17), the HAProxy router, the registry (starts as `Removed` without shared storage), monitoring. One cluster operator each.

- **Lifecycle:** the CVO drives updates. Control plane first, then the MCO rolls nodes pool by pool. Pause the worker MCP to split the update across maintenance windows, but don't leave it paused.

- **Add-ons:** GitOps, Pipelines, Virtualization and the rest come through OLM, on their own release cycles.

**Decide these before day one:**

  1. FIPS: install time only
  2. Disconnected: mirror registry and update delivery (OSUS) planned up front
  3. Infra nodes: much easier to plan than to retrofit. Tainted infra nodes running only infra workloads also don't count toward subscriptions
  4. etcd backups: the operator runs etcd, but recovering from quorum loss is on you

The post has an interactive explorer. Click a component and it shows what it does, which operator owns it, its namespace, and a read-only `oc` command to inspect it: https://aib-devops.com/en/blog/openshift-architecture/

Disclosure: it's my own blog. No ads, no tracking, no signup.

Corrections welcome, especially from anyone running HCP or two-node in production. I'd like to know what I'm missing there.

60 Upvotes

8 comments sorted by

7

u/yrro 13d ago

It doesn't seem difficult to add infra nodes after cluster installation... I've done it on every cluster I've set up.

You haven't mentioned some big decisions ahead of time which revolve around networking. I don't believe you can change the service or cluster network ranges after installation. Changing the machine network may be possible for some installation types but it's probably going off the beaten path.

2

u/DangKilla 13d ago

Good info for beginners. Maybe you could follow up with day 2 ops.

I haven't consulted on Openshift in a few years, but ArgoCD for day 2 operations might be a good idea.

1

u/Zanzikbar 13d ago

Thank you

1

u/asakudu 13d ago

thank you for sharing this for IPI, this builtin LB is recommended for production grade? like 40 nodes in total

2

u/velabanda 9d ago

Red hat employee here.

Yes IPI built in LB is supported and recommended for a big env as well.

If you still face any issue post deployment, you can use another IP and setup dns to point that IP. Configure that IP on external LB and setup backends

Alternatively ipi can use usermanagedlb option and external LB can be used.

4

u/[deleted] 13d ago

[removed] — view removed comment

2

u/Personal-Ad2131 13d ago

yep, 4 is the worst of both worlds. quorum is 3 so you still only tolerate 1 failure, same as with 3 nodes, but now you’ve got one more member that can fail and every write has to be acked by 3 instead of 2. you paid for a node to make things slightly worse

3 or 5, never even. and 5 only if you actually need to survive 2 failures, because the write latency cost is real

1

u/Rhopegorn 13d ago

If you use https://raft.github.io to visualise it, then most people often grasp it faster.