Automatically regenerate Tailscale TLS certs using systemd timers

In April of this year I wrote the Access local Nextcloud with HTTPS anywhere by using Tailscale TLS certificates blog post in which I showed how I am using Tailscale-generated certificates to access my locally hosted services with HTTPS. Since that time I have been happily using Nextcloud, Forgejo, and other apps via Tailscale with that glorious padlock visible in the browser address bar.

But today I could not access Nextcloud from my phone, the connection was erroring out. Further investigation showed that the TLS certificate generated by Tailscale expired. In contrast to what for example Certbot is providing, Tailscale certificates do not auto renew. And that is understandable, Tailscale is first and foremost a VPN service, and providing TLS certs is a secondary service for them.

I regenerated the certs manually, but that made me think, how to automate this task?

A quick Startpage query (still my favourite search engine) returned this Reddit thread and I thought it will be a perfect solution, but I will take it to 11. And by 11 I mean being notified whether the task finished successfully or not using using ntfy.sh.

I haven’t yet wrote a blog post about ntfy, but one day probably I will, because it is an awesome piece of software. For now, in short, it is a FOSS pub-sub notification service, and I have it self-hosted on my secondary VPS. I also have its app installed on my mobile phone to receive notifications on the go. I use it for a few tasks, like informing that the Borg backup of my main VPS finished without errors. The service page is publicly available, but the topics are private and only accessible for me.

Going back to the regeneration of TLS certs, for this task I am using systemd timers because I like systemd and I think this task is a good exercise in configuring its timers functionality, but if you prefer cron, then sure, cron will also be a right choice.

Everything below is done in the LXC container hosting my NextcloudPi instance, but is generic enough to be applied to any machine, virtual or not, that is using Tailscale certs and a HTTPS-capable server like Apache or nginx (I never know whether to capitalize nginx and need to check it every time).

I started with creating a bash script that regenerates the certificates, sends a notification informing that the job finished or errored out, and finally restarts Apache. Super basic Bash stuff.

/etc/certs/script.sh

#!/bin/bash

tailscale cert TAILSCALE_MACHINE_URL && curl \
   -H "Authorization: Bearer NTFY_ACCESS_TOKEN" \
   -d "Tailscale certs updated for NextcloudPi container" \
   https://notifications.stfn.pl/ntfy || curl \
   -H "Authorization: Bearer NTFY_ACCESS_TOKEN" \
   -d "Failed to update Tailscale certs for NextcloudPi container" \   
   https://notifications.stfn.pl/ntfy && \   
   systemctl restart apache2

The script first runs the tailscale cert command which generates new TLS certs for the provided URL. If that command finishes without errors, the part after && is run, which is a curl request to my ntfy.sh server with a notification saying that everything went fine. If the Tailscale command errors out, the “or” (||) part is triggered, which is, again, a notification for me, but with an error message. I had no idea that bash provides such ternary operators, such a cool new thing to learn. Finally Apache is restarted to load the new certificates.

Then I created a systemd service:

/etc/systemd/system/tailscale-certs.service

[Unit]
Description=Update Tailscale TLS certificates
After=network.target 

[Service]
WorkingDirectory=/etc/certs/
ExecStart=/usr/bin/bash /etc/certs/script.sh

[Install]
WantedBy=multi-user.target

One important thing here is the WorkingDirectory line. The tailscale cert command creates the certificate files in the directory when it is run, so we need to define the directory for the service. The outcome of this service will be two cert files generated in the /etc/certs/ directory, and that location must correspond with the certs location defined in the Apache configuration. Consult my linked blog post for details.

Also the script must be executable (chmod a+x /etc/certs/script.sh).

With the script and the service, the last part of the puzzle is creating the timer. A great resource on the topic is once again the Arch Wiki, and this entry has been the main source of information for me.

It is important that the service and the timer share the same filename, this is how they recognize each other.

/etc/systemd/system/tailscale-certs.timer

[Unit]
Description=Update Tailscale certs twice a month

[Timer]
OnCalendar=*-*-10,20

[Install]
WantedBy=timers.target

The only interesting part here is the

OnCalendar=*-*-10,20

which defines that the timer has to run twice a month, on the 10th and 20th day of the month. Not providing the time means it will run at midnight UTC.

I chose “twice a month” because I think it’s a reasonable time span ensuring that the certs will always be up to date, without flooding the certs service with too many requests.

Now the last thing to do is enable and start the timer by running

systemctl enable --now tailscale-cert.timer

systemd also provides a command to show the list of active timers, running

systemctl list-timers

should show the new tailscale-cert timer and the time left until the next trigger.

And that’s it. I just need to repeat those steps for every container I am running, and my services should be safe from expired TLS certs. And along the way I learnt something about bash and systemd. I’ve been using Linux since 2006 and I learn new things all the time and still feel I’ve just scratched the surface.

Thanks for reading!

If you enjoyed this post, please consider helping me make new projects by supporting me on the following crowdfunding sites: