Dynamic SSL certificates using LetsEncrypt on OpenShift

This post was originally published on the ETI blog here.

Managing SSL certificates in OpenShift can be a bit of a chore, especially when you have more than a few routes to manage. Having an automated mechanism to manage this helps with the operational overhead, and in this example LetsEncrypt is the weapon of choice.

You could quite conveniently use a wild card certificate to cover most of your routes but that doesn’t cover every use case that you might have, especially when you manage multiple domains. Consider also that wildcard certificates are deprecated[1] in favour of tooling that can provide programmatic access to easily create and renew SSL certificates on demand. There are a bunch of advantages (and disadvantages) to this and a tonne of articles out there, already covering the nuts and bolts of that topic, so I’m going to skip over that and instead share my experience deploying and using LetsEncrypt on OpenShift.

LetsEncrypt has been around for a while now and has been adopted into many environments so I thought it is about time that I shared how I have applied Lets Encrypt to solve my problem managing certificates across multiple domains on my OpenShift cluster.


Automated SSL Certification Authority (LetsEncrypt)

Lets Encrypt is a free, automated and open certification authority based on the ACME standard and is a service operated by the Internet Security Research Group and provides free, secure SSL certificates to the world. Lets Encrypt uses the ACME protocol to distribute certificates using tooling to manage the request, issuance, renewal and expiration of certificates.

Initially I thought about using a similar approach to Mak in his blog post covering the usage of the ACME client, and I have for another use case.  He describes using an ACME client to request a wildcard certificate using the tool at the cli, and then to update the certificate for the default route certificate on the OpenShift router. This is very convenient for use cases where you predominantly serve routes that are children of the same subdomain – for example *.apps.example.com. This is useful for tools like Red Hat CodeReady Workspaces/Eclipse Che which produce routes based on this pattern. With this method you do still need to craft a process (or remember) to renew the certificate and inject your freshly minted renewed certificate into the router. This led me on to looking at one of the suggestions in the comments of that article, the openshift-acme project on GitHub by Tomáš Nožička.

The OpenShift ACME Operator

The project that I used to automate the issuance of my certificates, provides an operator which once deployed into your cluster, watches for annotations on routes, and issues certificates according to those annotates across all namespaces. The installation of the Operator was very easy, and I followed the process for a ‘live cluster-wide’ implementation.

We will be deploying 2 components to make this happen:
– an Operator
– an annotation on the route

To deploy the operator into your cluster, run:

$ oc new-project prod-letsencrypt
$ oc create -fhttps://raw.githubusercontent.com/tnozicka/openshift-acme/master/deploy/letsencrypt-live/cluster-wide/{clusterrole,serviceaccount,imagestream,deployment}.yaml
$ oc adm policy add-cluster-role-to-user openshift-acme -z openshift-acme
Running the deployment gives us an operator deployed into the prod-letsencrypt namespace

Great, so you now have a new pod running in your cluster which creates SSL certificates on demand, how do you use it? Easy!

Issuing a certificate

Start off by deploying an application. If you haven’t got something already, the CakePHP application template is a good place to start.

oc new-project example-project
oc new-app cakephp-ephemeral
Create a secure route

To issue a certificate simply add this annotation to your route:

    kubernetes.io/tls-acme: "true"
Adding the annotation to the route to trigger certificate generation

What happens next is that a path-based route is created along side the route which you are requesting a certificate for. Something like /.well-known/ib78by87oby78y8788which LetsEncrypt validates using a HTTP-01 challenge.

Note: DNS-01 challenges are required for wildcard certificates, which are currently not supported by this operator.

The validation route is created and removed again, once validation passes

The freshly generated validation route will be removed once the CA has processed the request and the certificate has been successfully installed.


There are currently a couple of limitations with this Operator, but these wouldn’t stop me from using it in anger.

1) Validation routes are created on the default router which means that if you run your routers in a sharded mode, for example where you have a separate router to handle internal traffic (think management and development workloads).

Sharding in this way means that Lets Encrypt is unable to validate your ownership and issue the certificate. It’s worth noting that at the time of writing there is a PR in progress to resolve this issue: (https://github.com/tnozicka/openshift-acme/pull/69).

2) There is also not yet support for wildcard SSL certificates, but you can also follow development of that feature here (https://github.com/tnozicka/openshift-acme/issues/61). This is a limitation of the validation mechanism that is being used for routes.


So there you have it, it’s actually really simple to deploy the LetsEncrypt operator and use it t0 generate SSL certificates on demand. If you’re wondering, the certificates are automatically re-issued when they get close to their expiry too! It’s always a treat when you can reduce some of your operational overhead.

Mak’s post about using the ACME client: https://blog.openshift.com/create-https-based-encrypted-urls-using-routes/
More information about LetsEncrypt: https://letsencrypt.org/how-it-works/
The foundation behind LetsEncrypt: https://www.abetterinternet.org/

[1] RFC 6125: https://tools.ietf.org/html/rfc6125#section-7.2
[2] Wildcard SSL certificates with FreeIPA: https://frasertweedale.github.io/blog-redhat/posts/2017-06-26-freeipa-wildcard-san.html
[3] Wildcard certificates with FreeIPA: https://www.freeipa.org/page/Howto/Wildcard_certificates
[4] Lets Encrypt ACME challenge mechanisms: https://letsencrypt.readthedocs.io/en/latest/challenges.html#dns-01-challenge


Connect with Red Hat Services

Learn more about Red Hat Consulting
Learn more about Red Hat Training
Learn more about Red Hat Certification
Join the Red Hat Learning Community
Subscribe to the Training Newsletter
Follow Red Hat Services on Twitter
Follow Red Hat Open Innovation Labs on Twitter
Like Red Hat Services on Facebook
Watch Red Hat Training videos on YouTube
Follow Red Hat Certified Professionals on LinkedIn

  1. Great article! I was going to say about Limitation 2) it is possible to generate wildcard certs with certbot/letsencrypt, I do it every 3 months. I did it a few days ago. You just have to point it to an updated server like this one: https://acme-v02.api.letsencrypt.org/directory

    # get latest certs from certbot from DNS challenges.
    sudo certbot -d example.com -d *.example.com -d *.apps.example.com -d example.org -d *.example.org -d *.apps.example.org –manual –preferred-challenges dns certonly –server https://acme-v02.api.letsencrypt.org/directory

    # generate my server.jks java keystore.
    sudo cp /etc/letsencrypt/live/example.com-0001/privkey.pem /srv/example.com/server.key
    sudo cp /etc/letsencrypt/live/example.com-0001/fullchain.pem /srv/example.com/server.crt
    sudo chown $USER: /srv/example.com/server.* /srv/example.com/root.crt
    cat /srv/example.com/root.crt /srv/example.com/ca2.crt /srv/example.com/server.crt > /srv/example.com/merged.crt
    openssl pkcs12 -export -in /srv/example.com/merged.crt -inkey /srv/example.com/server.key -out /srv/example.com/server.p12 -name example.com
    keytool -importkeystore -srckeystore /srv/example.com/server.p12 -destkeystore /srv/example.com/server.jks -srcstoretype pkcs12
    chmod 700 /srv/example.com/server.* /srv/example.com/root.crt
    cp /srv/example.com/server.* /srv/computate.org/

    # restart any local web services pointing to the certs
    sudo systemctl restart httpd-2.4.service haproxy.service keycloak-3.3.0.service
    sudo systemctl status httpd-2.4.service haproxy.service keycloak-3.3.0.service

    # update openshift with certs.
    sudo -i
    oc export secret router-certs > ~/old-router-certs-secret.yaml
    oc create secret tls router-certs –cert=/srv/example.com/server.pem –key=/srv/example.com/server.key -o json –dry-run | oc replace -f – –config=/var/local/oc-3.9/host-config/master/admin.kubeconfig
    oc annotate service router service.alpha.openshift.io/serving-cert-secret-name- service.alpha.openshift.io/serving-cert-signed-by- –config=/var/local/oc-3.9/host-config/master/admin.kubeconfig
    oc annotate service router service.alpha.openshift.io/serving-cert-secret-name=router-certs –config=/var/local/oc-3.9/host-config/master/admin.kubeconfig
    oc rollout latest dc/router -n default –config=/var/local/oc-3.9/host-config/master/admin.kubeconfig

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.