My Journey to Kubernetes onto Bare Metal — Part 5: Cert-Manager

9 min readMay 18, 2020

Automatically creating SSL certificates the kubernetes way.

The Problem

This is one of the most contentious aspects of containers: How to get your application an SSL/TLS certificate in order to be trusted on the internet by you or your users. The best security is only as strong as it’s weakest point. So how can an individual attain such a valuable product with spending as little as possible. In comes Let’s Encrypt, BuyPass and others that offer free certificates for a specified amount of time. Remember, this guide is how to get things done to provision your kubernetes application as auto-magically as possible. So now let’s find out how to get it working.

HTTPS or DNS validation

There is a framework called ACME that follows certain protocols and process to request a certificate in an automated fashion. The whole reason for a certificate is to prove to browsers that you are who you are and that the website that is being connected to can be trusted. In order to verify that you actually own the URL in question is to do either:

A) Cert-Manager will write a small file with a secret message in it on the actual web server. Then the certificate generating system (Let’s Encrypt or BuyPass) will wait a while and then attempt to read the contents of that file and make sure it can be reached and read the secret code it was instructed to write there. If everything in the process chain works then you get your certificate!

B) Make sure the DNS entry you have is created. You give cert-manager the credentials to go create a special DNS entry at your DNS registration site which will contain some secret content. Then after a while the certificate generating system (Let’s Encrypt or BuyPass) will go read the contents of the DNS record and validate with what it was expecting. This is to ensure that you own the domain that the certificate is being created against.

I chose to use DNS validation since I do not have to ensure that a website is available.

Preparing for DNS Validation

There are a number of DNS services that are compatible with cert-manager. Here is a guide on how to configure DNS to use with cert-manager. I’m going to show you how I completed this task.

Create a Service Account

Credentials must be created in order for cert-manager to create a DNS record. I am assuming that you have a Google Cloud account and can get to your administrative dashboard.

Step 1: Create a new project. Click on the project selector for a pop-up to appear that will let you create a new project.

Step 2: Switch to the project and then navigate to “Service Accounts”:

Step 3: Click on “+ Create Service Account”

Step 4: Fill in the details of the new service account. Give it any name you prefer. The ID will automatically be filled. Click the “Create” button when done.

Step 5: A role must be assigned. Click on the drop-down arrow.

Step 6: A pop-up will appear for you to select a role. Select “DNS” — > “DNS Administrator”

Step 7: Click on the “Continue” button when done.

Step 8: This is optional. You may grant users to use this service account. When done we must download a private key that we have to import into Kubernetes. Please click on the “+ Create Key” button.

Step 9: Leave the default “JSON” selected and click on the“Create” button.

Step 10: A json file will begin to be downloaded and this notification will appear. Click on “Close”.

Step 11: You are done. Please click the “Done” button.

Create a namespace

Create a namespace for this project with the command:

kubectl create namespace cert-manager

Now let’s try to install cert-manager. Notice that we are not using Helm nor downloading a file. I used this guide, but I could have used a Chart if I chose to do so.

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.0/cert-manager.yaml

If I want to remove the installation then just execute:

kubectl delete --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.0/cert-manager.yaml

Import the DNS JSON Private Key

In step #11 above we had downloaded a .json file that contains the information about the service account that has permissions to modify the DNS records on Google Cloud DNS. We must import that file as a secret into the kubernetes namespace for cert-manager. I executed the command below to import the file “mp-gcdns-key.json” as a secret named “clouddns-dns02-solver-svc-acct”.

kubectl create secret generic clouddns-dns01-solver-svc-acct --from-file=mp-gcdns-key.json -n cert-manager

The imported file will look like this:

Create the Cluster Issuer

The next step is to create a Cluster Issuer that will actually go out and retrieve the certificate. This configuration file is pointed to Let’s Encrypt, but I also have an issuer pointed to BuyPass by only changing the server URL.

Import the yaml file below and now you have a Cluster Issuer created to make use of the service account.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: le-clusterissuer-prod
  namespace: cert-manager
spec:
  acme:
    email: Richard.Nunez@macrossplusinc.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: le-clusterissuer-prod
    solvers:
    - dns01:
        clouddns:
          # The ID of the GCP project
          project: my-project-123456
          # This is the secret used to access the service account
          serviceAccountSecretRef:
            name: clouddns-dns01-solver-svc-acct
            key: mp-gcdns-key.json

To validate that the Cluster Issuer is ready, type the command:

kubectl describe clusterissuer le-clusterissuer-prod -n cert-manager

And a description will print out to the screen. Pay close attention to the last few lines that show the “Status” of the issuer:

Status:
  Acme:
    Last Registered Email:  Richard.Nunez@macrossplusinc.com
  Conditions:
    Last Transition Time:  2020-03-26T17:20:10Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

This means you are ready to request certificates.

Requesting a Certificate

There are two methods of requesting a certificate: Create a certificate object manually or add annotations to a service that will do it for you.

Request using the Certificate Object

In order to manually request a certificate you must create a certificate object along with describing the DNS entry you want to create it for. Below is the yaml that will create that object. This example is for the “whoami” deployment (which consists of a service & deployment object) within the “whoami” namespace.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: whoami-cert
  namespace: whoami
spec:
  commonName: whoami.macrossplusinc.com
  secretName: whoami-cert-prod
  dnsNames:
  - whoami.macrossplusinc.com
  issuerRef:
    name: le-clusterissuer-prod
    kind: ClusterIssuer

metadata is just the arbitrary name I gave this certificate object, called “whoami-cert” within the “whoami” namespace
“commonName” is a common name to be used on the Certificate. If no CommonName is given, then the first entry in “DNSNames” is used as the CommonName.
“secretName” is the secret to store the retrieved certificate from Let’s Encrypt.
“dnsNames” is a list of DNS entries that the certificate will be valid against. These DNS entries must already exist in your registrar, which in my case is Google Cloud DNS.
“issuerRef” is the Cluster-Issuer that we set up above that will retrieve the certificate from Let’s Encrypt. The best practice is to use the staging server, but I am showing you the production url.
Under “issuerRef” is pointer to the Cluster-Issuer, so I define the “name” and “kind”. Read about them here. There are two kinds of issuers: Cluster and local. Cluster-Issuers are available for the whole kubernetes cluster. And regular issuers, named “Issuer” have a scope to the local namespace it resides within.

Check if successful

After I have applied the above configuration in order to create a certificate request I can check the status with the command:

kubectl describe certificate whoami-cert -n whoami

Look at the bottom for the “Status” section. It will take a short while to update the status. If you look at your DNS records dashboard during this process you will see a TXT record created with some text content. This content is what Let’s Encrypt is looking for in order to validate that the DNS is owned by you.

Let’s run the describe command again and see the status of the certificate request:

Spec:
  Common Name:  whoami.macrossplusinc.com
  Dns Names:
    whoami.macrossplusinc.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       le-clusterissuer-prod
  Secret Name:  whoami-cert-prod
Status:
  Conditions:
    Last Transition Time:  2020-03-26T17:28:21Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2020-06-24T16:28:21Z
Events:                    <none>

This shows the spec and the “Status” of the certificate, which was a success. If we go into the kubernetes dashboard within the “whoami” namespace we can find the create secret:

Clicking on “whoami-cert-prod” will show us:

Notice that the tls.crt and tls.key are data in them? If this process had failed then tls.crt would be ZERO bytes. We are done retrieving the certificate manually!

To make use of this certificate you must modify your Ingress object to specify the secret that contains your certificate:

tls:
    secretName: whoami-cert-prod

Using Annotations

The other method to retrieve certificates is to use annotations. This is great because this automates the creation of the certificate object and also is used within Helm deployments. There is where automation helps.

I am going to be using my “whoami” deployment again for this example.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: whoami-ingress
  namespace: whoami
  annotations:
    cert-manager.io/cluster-issuer: le-clusterissuer-prod
spec:
  tls:
  - hosts:
    - whoami.macrossplusinc.com
    secretName: whoami-cert-prod
  rules:
  - host: whoami.macrossplusinc.com
    http:
      paths:
      - path: /
        backend:
          serviceName: whoami
          servicePort: 80

The deployment of the Ingress object above will create an ingress route to the service “whoami” and at the same time request a certificate be requested and stored in the secret “whoami-cert-prod”. You can check your kubernetes dashboard for the success of this process.

Conclusion

This guide revealed how I installed, configured, and implemented Cert-Manager to generate certificates for my services that I want to expose to the world. There are some diagnostic commands I can give you to help troubleshoot so I will add them later.

Part 6: ExternalDNS