January 6, 2023

Autocert Introduction and Tutorial

Posted on January 6, 2023  •  7 minutes  • 1393 words

Issue and renew TLS/HTTPS certificates in Kubernetes.

If you want to secure TLS/HTTPS communication in a Kubernetes cluster, you have multiple options:

Autocert provides a simple solution for cluster-internal mTLS. It handles certificate issuing and renewal and comes with its own certificate authority (Step CA). With Autocert, you just add an annotation to the resources that need certificates and Autocert does the rest. It issues the certificates, stores them in a volume mount and takes care of the renewal.

How Autocert Works

Here is a big picture of Autocert:

autocert big picture

Autocert provides a mutating admission webhook and a certificate authority based on Step CA. The webhook intercepts and patches pod creation requests with some YAML to inject a bootstrapper container and a renewer sidecar that handle the obtaining and renewing of certificates from the certificate authority.

Autocert should be deployed in a separate namespace which is commonly named step. After installation, you will find the following pods and services in this namespace:

For bootstrapping, autocert uses the one-time token bootstrap protocol from Step CA to mutually authenticate a new pod with the certificate authority, and obtain a certificate. This goes like this:

  1. The one-time token is generated by the admission webhook and transmitted to the injected autocert-bootstrapper container via a Kubernetes secret.
  2. The autocert-bootstrapper container uses the one-time token to obtain a certificate at the certificate authority.

For renewal, the autocert-renewer sidecar simply uses mTLS with the not-yet-expired certificate for mTLS with the CA. Thus a valid certificate is required at all times to keep the service accessible. For example, if you stop a training cluster and restart it after the certificate has expired, you need to run the deployment, again to reinitialize the process.

Following containers are injected to the pods:

The certificate, the private key and the CA certificate are monted at /var/run/autocert.step.sm/.

Enabling and Configuring using Labels and Annotations

Autocert is enabled on per namespace basis with the following label:

kubectl label namespace default autocert.step.sm=enabled

Autocert certificate issuing and renewal is configured on a pod level with these annotations:

autocert.step.sm/name: hello-mtls-1h.default.svc.cluster.local
autocert.step.sm/duration: 1h
autocert.step.sm/owner: "999:999"
autocert.step.sm/mode: "0600"

Only the name annotation is required. All other annotations are optional.

Application configuration is out of scope. Autocert only does the certificate handling. You need to take care that your application actually makes use of the certificates and enforces mTLS. This also means that servers need to be reloaded on certificate renewal.

Kubernetes RBAC

Autocert needs a cluster-wide permission to create and delete secrets. In effect any pod in the cluster can get certificate with any identity. You may ask: How is that secure? Well, you can secure it on a higher level:

  1. Don’t give people kubectl access to your production clusters
  2. Use a deployment pipeline based on git artifacts
  3. Enforce code review on those git artifacts

If you find yourself in a position where you are trying to implement more fine grained controls you should consider using a service mesh. For example, if you want a service to accept only connections from clients in a certain namespace, you can certainly configure the service to do so, but this quickly gets cumbersome. Moreover, you basically implement the authentication aspect in potentially multiple different ways depending on the applications you deploy. With a service mesh, you can do this in a consistent and unified way.

Commands for Management and Troubleshooting

Get the PKI and Provisioner secrets (adjust the namespace as necessary):

kubectl get -n autocert \
  secret/autocert-step-certificates-ca-password \
  -o jsonpath='{.data.password}' | base64 --decode
kubectl get -n autocert \
  secret/autocert-step-certificates-provisioner-password \
  -o jsonpath='{.data.password}' | base64 --decode

The initial configuration is done with a Kubernetes batch job. You can inspect its logs to get details such as the CA URL and the root certificate fingerprint:

kubectl -n autocert logs job.batch/autocert

Delete the configuration job:

kubectl -n autocert delete job.batch/autocert


This tutorial gives a quick introduction to autocert. You will learn

You will need git, minikube, kubectl and Helm to follow this tutorial. Let’s go!

  1. Create a test cluster.
minikube start autocert
  1. Check that the admission webhook is enabled.
kubectl api-versions | grep "admissionregistration.k8s.io/v1"
  1. Create a cluster role binding for the default service account with role cluster-admin.
kubectl create clusterrolebinding autocert-init-binding \
  --clusterrole cluster-admin \
  --user "system:serviceaccount:default:default"
  1. Install autocert with Helm.

Helm is used as it is the recommended way of installing autocert.

helm repo add smallstep https://smallstep.github.io/helm-charts/
helm install autocert smallstep/autocert \
  -n autocert --create-namespace
  1. Enable Autocert for a namespace.

You need to enable autocert per namespace by adding the label autocert.step.sm=enabled to the namespace. In this case we enable it for the default namespace and check if it is enabled.

kubectl label namespace default autocert.step.sm=enabled 
kubectl get namespace -L autocert.step.sm
  1. Clone the examples repository and change into the autocert directory.

This contains a client and server example. The server is based on Nginx, the client uses curl.

git clone https://github.com/mitja/cozykube-examples
cd cozykube-examples/introductions/autocert
  1. Inspect the deployment manifests x and y. Note that they contain the annotation autocert.step.sm/name.
cat nginx/hello-mtls.server.yaml | grep autocert.step.sm
cat curl/hello-mtls.client.yaml | grep autocert.step.sm
  1. Take a look at the certwatch.sh file.

Note that there is a reloader script. This is necessary to reload Nginx after new certficate has been issued on renewal.


while true; do
    inotifywait -e modify /var/run/autocert.step.sm/site.crt
    nginx -s reload
  1. Build and deploy the server and check that everything is working by inspecting the rollout status.
cd nginx
minikube -p autocert image build \
  -f Dockerfile.server \
  -t hello-mtls-server-py-gunicorn .
kubectl apply -f hello-mtls.server.yaml
kubectl rollout status hello-mtls
cd ..
  1. Build and deploy the client
cd curl
minikube -p autocert image build \
  -f Dockerfile.client \
  -t hello-mtls-client-curl .
kubectl apply -f hello-mtls.client.yaml
cd ..
  1. List the certs of the server to see that they have been issued.
export HELLO_MTLS=$(kubectl get pods \
  -l app=hello-mtls \
  -o jsonpath='{$.items[0].metadata.name}' \
kubectl exec -it $HELLO_MTLS -c hello-mtls -- 
  ls /var/run/autocert.step.sm
  1. Connect from the curl client using the issued certificate and certificate authority cert.
export HELLO_MTLS_CLIENT=$(kubectl get pods \
  -l app=hello-mtls-client \
  -o jsonpath='{$.items[0].metadata.name}' \
kubectl exec $HELLO_MTLS_CLIENT -c hello-mtls-client -- \
  curl -sS \                                                                                       
  --cacert /var/run/autocert.step.sm/root.crt \
  --cert /var/run/autocert.step.sm/site.crt \
  --key /var/run/autocert.step.sm/site.key \
  1. Provoke an error by connecting from the client without presenting its certificate. The server should reject the connection.
kubectl exec $HELLO_MTLS_CLIENT \
  -c hello-mtls-client -- curl -sS \
  --cacert /var/run/autocert.step.sm/root.crt \
  1. Clean up by deleting the cluster
minikube delete -p autocert

This concludes the introduction tutorial to Autocert. In the tutorial you have seen the basic use case of autocert of mTLS in a cluster.

What else can you do with Autocert?

Here are some other Autocert use cases:

Summary and Where to Find More Information About Autocert

Autocert is a simple and basic building block for TLS certificate issuing and renewal for Kubernetes pods. It is can be a solution for implementing cluster-internal mTLS and can also federate to secure communication with cluster-external resources. It is not as feature-rich as a service mesh but also much easier to understand and manage. Here are links to more information about Autocert: