And the CI with many-GB Docker images is very painful... Turning on layer caching usually makes the process even slower as it needs to pull and unpack the previous image before starting, and if you turn it off you're downloading a ton of deps on every build.

If you separate the heavy stuff into a base-image you still have to load it on the beginning of CI, which without beefy machines with local SSD caching can take a loooong time.