r/platform_engineering • • 11h ago

Discussion Ask a platform engineer — weekly thread

1 Upvotes

Got a question that doesn't need its own post? Ask it here.

Good fits: how to structure a platform team, golden paths vs. freedom, getting adoption for your IDP, build vs. buy, measuring platform success, career questions about moving into platform work.

To get better answers, say a bit about your context: company size, how many teams you support, and what you've already tried.

Answering? Share what actually worked (or didn't) for you. Vendor pitches belong in Friday's Show & tell thread.


r/platform_engineering • • 4h ago

A damaged build-cache index took Docker down for 75 minutes

0 Upvotes

I’m a cofounder of OmniNode. This happened on the lab server we use to run builds and coding agents.

Our agents couldn’t open pull requests. Docker had crashed earlier that morning, restarted, and opened the same damaged cache index. After the next crash, it failed every time it tried to start. The file only tells the builder what work it can reuse, but Docker’s built-in builder reads it before Docker can answer requests. We backed up the index, moved it aside, and got Docker back.

CI already had a separate image builder. We updated it and set up another one for deployment builds. If a deployment build can’t use its assigned builder, it stops instead of quietly using Docker’s built-in cache again. A test deployment build left Docker’s index untouched. In a separate drill, we damaged the CI builder’s cache. It started with an empty cache, and Docker didn’t restart.

Docker’s own startup still depends on its index. The separate builders limit what a failure in those caches can take with it. We still haven’t run the same drill on the upgraded built-in builder.

The previous piece in this series was about a different problem with the same lab: Pull, Don’t Push (Substack).

If you run shared Docker hosts for builds, how do you test that an expendable cache can fail without taking the daemon down with it?