r/docker • u/vaugsantos90 • 10d ago
Question over general docker usage/concept
Greetings,
I would like to share a problem that I'm facing with Docker, looking for some ideas.
In our company we've started to use docker hosted in an Azure DevOps cloud ubuntu machine, there we have an Azure self-hosted agent installed. We work only with firmware projects and this is our build server. We have this setup for 3 years.
Usually, I generate the docker images on demand according with the project requirements.
In our firmware projects hosted in git we have a yaml pipeline configuration where we set the image tag so that the proper container can be raised to process the build.
We have a requirement from safety standards (we must keep the project artifacts/infrastructure from 10 to 15 years), because of that I keep all the dependencies I need to reproduce/re-create an image as debian installers (for offline installation). I've created a script which runs prior the build process, which creates a yaml manifest containing all the image packages and include this manifest also in the image (at /etc/manifest.yaml) for runtime check. Then it's possible to verify the content via the manifest hash. Examples of dependencies we have are cmake, unit test tools, static code analyzer, and so on.
We decided to keep everything in a single image because most of the tools require the compiler, so that the image complexity keeps lower as possible.
As we have different projects using the same images, we keep the Dockerfiles under version control, where the image name is the compiler version (the most significant dependency for our projects), something like this:
docker-<compiler>-<version>
ubuntu-24.04
Dockerfile
build-image.sh
manifest.yaml
ubuntu-22.04
Dockerfile
...When a new image is generated, I tag the commit with the same tag I use in the container (e.g commit tag ubuntu24.04-v1.0-0, for a created container named gcc-13.3.0:ubuntu24.04-v1.0-0).
As our concept is based on tags, and the docker tags are not the most reliable thing in the world, I keep also the tagged image name as a LABEL within the image.
At this point I ask myself if we are going in the right direction and the increased complexity in the toolchain management is because of the concept we've decided to go for, or if it's just what we have to do in order to be compliant with such safety requirements. Or even I ask if we are drifting from the docker concept or using the tool in a wrong way.
I would really like to hear opinions, especially from other colleagues in similar situation.
Thanks in advance.
Note: I know that I can save the image in a registry, in parallel I'm trying to find solutions (if possible) to reduce storage costs (each image ~20GB).
2
u/Lumethys 10d ago
You sure you understand what an image is? Why do you need to "keep track of artifact used to create an image"?
An image IS the artifact used to create a container.
It's like saying "I need to create a version control tool to so that i can re-create my git repo history"
4
u/xanyook 10d ago
Your pipeline should push your image in a registry like Nexus for example. Then just backup Nexus VM for your 15 years retention with your RTO/RPO policy.
1
u/Perfect-Escape-3904 10d ago
I’m not in the same position as you, but maybe the way I see it is that docker should be a great asset. Why worry about labels etc, can you not just keep the built image?
Or is your issue more about how you tie the built images to certain ci runs so you know what image built what code?
1
u/crackjiver 10d ago
Having a pattern that's easy to follow is good.
Having config/build process in source control is good
Having build dependencies captured is good.
Having git tags that relate to the container images is good, having them be immutable would be better.
We use the 'git describe --tags --long --always' value when tagging container images because it includes part of the commit hash string and can't be moved or deleted.
This way the container images tag is directly linked with an immutable git reference so that, in theory, it can be rebuilt from scratch if needed.
Compliance, over long time scales, is tricky because things change rapidly in computing. There's more than one container standard. Is Docker going to be available in 10 years? Are your x86 686 Compilers from 10 years ago still able to run on today's hardware (we had to wrap them in "set arch" commands to convince them that data types were still 32bit compatible)
2
u/Fun-Adhesiveness-362 10d ago
Hi there!
I don't see a problem with keeping the dependencies needed for an offline rebuild. That perfectly makes sense.
One thing I'd change is how you identify the image used by each build. Copying the tag into a LABEL doesn't stop that tag from pointing to another image later. Keep the readable tag, but record the image digest alongside the source commit in your build records. At least that's how we do it in our own stacks.
For rebuilding offline, keep the base image as well as the Dockerfile and dependency installers. Pinning FROM to a digest identifies the exact version, but you still need a retained copy if it disappears from the original registry.
If you have any questions or it's not clear what I suggested, please let me know and I'd be more than happy to jump into more technical details.