Using Let's Encrypt for Internal Servers

When playing around with new software or developing new web sites SSL is something that is not only desired but is required and expected. In the past this has meant creating self signed certificates for local development machines and then requesting valid certificates for production.

This can cause issues however. Many applications allow you to connect to an API that has a self signed certificate, but an option has to be enabled to allow it to ignore SSL verification. For example Chef can be deployed using a self signed certificate but all the managed nodes have to have ssl_verify_mode :verify_none added to their configuration.

As this meant that there was more things that needed to be managed by the environment I was very pleased when Let’s Encrypt launched which allowed free SSL certificates. However it did not work immediately for internal systems because the challenge method needed to be able to access a website on a public address! I was not prepared to punch a hole through my firewall every 90 days. But now there is a DNS challenge for Let’s Encrypt.

Local Domain Name

The Let’s Encrypt DNS challenge works by asking you to add a TXT record to a publicly accessible DNS zone. At first I misunderstood this and thought it would mean my entire local DNS would have to be public - wrong. What it means is that the but that has the TXT record has to be public which is a subtle difference. This meant I needed to change my internal DNS name.

I run Bind for my local DNS and it originally had the domain name of turtlesystems.local. To be able to satisfy the requirements of the DNS challenge this would need to change as I was not able to buy a public domain name of turtlesystems.local. So I changed it to be a sub domain of my public one, which gave me home.turtlesystems.co.uk.

So now my internal DNS has the zone for this sub domain but also as it is part of turtlesystems.co.uk I can create TXT records for the DNS challenge as required. My internal names are not exposed to the internet and I am able to use Let’s Encrypt.

Obtaining a Certificate

The preferred tool for getting certificates is certbot. After I installed this it is used to request a certificate, e.g.:

$> certbot -d node1.home.turtlesystems.co.uk --manual --preferred-challenges dns certonly

This command will output the name of the DNS TXT record to create and its value and wait for confirmation:

Please deploy a DNS TXT record under the name
_acme-challenge.node1.home.turtlesystems.co.uk with the following value:

y8yrjF1l667du8YZlgy0wKNBlCnSaUndcrNmQL3vX6b

Once this is deployed,
Press ENTER to continue

As this is a new record in DNS there should be no propagation delay as no DNS server will have the record cached so the certificate will be obtained very quickly after confirmation of the TXT record being created.

The certificate will be stored on the local machine in the following location, assuming that node1.home.turtlesystems.co.uk was the record being created:

/etc/letsencrypt/live/node1.home.turtlesystems.co.uk

When certificates are renewed it is the symlinks in this directory that are updated. If using a web server with these certificates it is a good idea to link directly to these files without copying them.

More information on this can be found here.

DNS Provider

Whilst I was sorting out my certificates for machines I came across Traefik which is a modern HTTP reverse proxy and load balancer to deploy microservices with ease. It can use Lets Encrypt to get certificates and it supports the DNS challenge, however my DNS provider was not listed in the supported providers.

Route53 was listed as supported to I moved my domain over to Route53 and configured an IAM account which Traefik uses to control the challenge.

Moving the records over to Route53 was an easy enough task and then I had to change the nameservers for my domain registrar. The hardest part was to get the permissions right for the IAM account to create records. The IAM account access keys are passed to Traefik so it can create records on my behalf.

The permissions that were required are:

As can be seen from the screen shot this is a custom policy that I created as I could not work out the permissions from all the default policies.

Summary

This setup works very well and I am able to create any certificates for any machines or services that I create withing my home office network. By using Amazon Route53 I am able to automate the challenge side of things.

Share Comments