Provide guidance for using update-ca-certificates in distroless images

[lbussell]

Describe the Problem

Our Ubuntu Chiseled images only includes the ca-certificates_data slice, which excludes tools like update-ca-certificates in order to reduce image size (this utility isn't typically needed at container runtime). However, if users want to add certificates at container build time, there's no documented way to do so in Ubuntu Chiseled.

Describe the Solution

There should be a documented way to run the update-ca-certificates tool in the image's build layer, and copy the results to the runtime layer.

Other Information

Context: https://devblogs.microsoft.com/dotnet/announcing-dotnet-chiseled-containers/comment-page-2/#comment-20182

[lbussell]

Slightly related: #4997

[marcmognol]

Hi @lbussell

In my company, I do prefer mounting the certificate as a volumeMount instead of integrating it in the container image.

Below an example of my Deployment manifest :

          volumeMounts:
          - name: secrets
            subPath: ROOT-CA-CERTIFICATE
            mountPath: /etc/ssl/certs/contoso-ca-certificates.crt
            readOnly: true

The certificate is stored in a common keyvault and mounted as a volume in all my deployments. This way, I can update the certificate in the keyvault once, restart all deployments and voilà !

Let me know, if my answer is not clear.

[ladeak]

So you install the certificates with update-ca-certificates on a machine and mount the resulted file?

[marcmognol]

@ladeak

No, I don't setup the certificate on the machine with update-ca-certificates.
Our workloads are deployed in an Kubernetes cluster with AKS (Azure Kubernetes Services)

What I do:

• Extract the root CA certificate into a file
• Copy the content of this file into a keyvault,
• Mount the keyvault as a Volume in my kubernetes deployment with a SecretProviderClass

If you don't have such a system, you can put the certificate file in a folder and mount it into the container to the path I mentioned in my previous comment.

Marc

[ladeak]

I don't recall the exact problem, but my recollection says the above method did not work for all cases. I do use exactly the same approach, however some certs had to be 'installed'.

[jfheins]

We encountered the same issue. Chiseled images seem to work for us, save the custom certificate.
In our company, we add a company root CA into the container and execute RUN update-ca-certificates afterwards:

ADD --chmod=644  company_root_ca.crt /usr/local/share/ca-certificates/company_root_ca.crt
RUN update-ca-certificates

This is used for interfaces to other services within the company. So another team has a HTTPS endpoint that uses this internal cert and we want to do HTTPS requests to that service.
Thus (if I'm not mistaken) the running application needs this cert in the cert store.

Is there a way to make this work in a chiseled container?

[richlander]

Related: richlander/container-workshop#5 (comment)

[richlander]

You can also run update-ca-certificates in muti-stage-build.

[jfheins]

I think I got it to work and I'm happy to share my solution. Previously I had a Dockerfile like this:

FROM mcr.microsoft.com/dotnet/sdk:8.0-bookworm-slim AS build
COPY . /src
RUN dotnet publish "/src/App.csproj" -c Release -o /app

FROM mcr.microsoft.com/dotnet/aspnet:8.0-bookworm-slim AS final
# Some stuff
COPY --from=build /app .
ENTRYPOINT ["dotnet", "App.dll"]
# Add certificates
COPY --chmod=644 company_root_ca.crt /usr/local/share/ca-certificates/company_root_ca.crt
RUN update-ca-certificates

Now, if I change the second stage to chiseled, the update-ca-certificates shell script won't work.
It was already multistage, so just "use a multi-stage build" didn't help me here.
The solution looks like this:

FROM mcr.microsoft.com/dotnet/sdk:8.0-bookworm-slim AS build
COPY . /src
# Use shell utils here as they are not available in chiseled
COPY --chmod=644 company_root_ca.crt /usr/local/share/ca-certificates/company_root_ca.crt
RUN update-ca-certificates
RUN dotnet publish "/src/App.csproj" -c Release -o /app

FROM mcr.microsoft.com/dotnet/aspnet:8.0-jammy-chiseled-extra AS final
# Some stuff
COPY --from=build /app .
ENTRYPOINT ["dotnet", "App.dll"]
# Add certificates
COPY --from=build /etc/ssl/certs /etc/ssl/certs

I guess the important bit is to know which folders are modified by running the tool.
As long as the tools you want to run just modify files and you know which files in which folders are modified, one can execute them in the SDK image and then copy over the folders from there.

[ladeak]

I think the key would be @richlander and @jfheins to find a solution where the certificates are not baked into the image. That seems to be rather an anti-pattern.

[jfheins]

Baking in can work fine if the lifetime of the cert is way longer than your release cycle.
If that is not the case, it could be an option to mount /etc/ssl/certs as a volume and modify it from outside.

[richlander]

Not a big deal, but it probably makes sense to use one distro for both stages. You are using a combination of Debian and Ubuntu.

There are two options:

• Use multi-stage-build (per the example above).
• Use an offline build (can still be in containers) and then volume mount result from the host.

I'm assuming the company-root-ca.crt isn't actually a certificate, but a list of certificate thumbprints. Per my understanding, it shouldn't itself be copied to /etc/ssl/certs but for those thumbprints to be copied into /etc/ssl/ca-certificates.crt via update-ca-certificates.

The volume mounting approach is the most secure and reliable.

Context: https://manpages.ubuntu.com/manpages/xenial/man8/update-ca-certificates.8.html

[lbussell]

Potentially related: dotnet/aspnetcore#56582

[mthalman]

This should be fixed by canonical/chisel-releases#266 in the next Chisel release.

[lbussell]

This should be fixed by canonical/chisel-releases#266 in the next Chisel release.

Are you saying that we should add the ca-certificates_bins slice in the middle of a servicing release? It also includes other binaries that aren't needed - openssl_bins and sed_bins. I'm not sure we need to add this.

[mthalman]

I'm not making any statement about servicing releases. Just indicating that a fix for this is incoming from Chisel.