CI

what and why At $job, we are using jenkins with a dozen or so docker runners, each running their own docker daemon. At least a few hundred builds are carried out daily on these runners, mainly producing docker images that are then pushed onto a testing/production container registry. But two jobs from the same repo might not be using the same CI runner (most of the time they are not). It means each docker runner might need to rebuild the first couple of layers from a given Dockerfile, despite another worker having already built the same layers a few hours/minutes ago. That’s wasting CPU cycles, bandwith, and it’s also increasing the time each build is taking. ...

what & why a bit of context At $dayjob, our backend stack is 99% python based, and deployed through containers on k8s clusters. That means that a lot of time spent in our CI is downloading and installing librairies and modules from Pypi, the Python Package Index. For the sake of data sovereignty and integrity, our CI workers are deployed on-prem, using Gitlab Runners. But due to the variety of projects, we are depending on hundreds of various libraries, ranging from fastapi/flask and their related libraries, to more esoteric ones like pysnmp, yang, scrapli, netmiko/paramiko.. ...