This guide is published under the Creative Commons Attribution-ShareAlike 4.0 International license.

You are free to:

for any purpose, even commercially.

Under the following terms:

Read more on https://creativecommons.org/licenses/by-sa/4.0/

install-kvm-host.sh is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This guide is an Asciidoc document. It is hosted on Github and can be read online (or downloaded) as HTML or PDF document:

install-kvm-host.sh is distributed together with the Asciidoc sources in the github repository.

Born in 1972, Dirk Hillbrecht started working with computers in 1986, when his school offered a programming course using Turbo Pascal 3.0 on Apple II computers. In the 1990s he studied "Mathematics with focus on Computer Science" in Hannover. He administrated networks of Sun and Silicon Graphics machines at the university and witnessed the raise of Linux almost from its beginnings. Since 2000, he writes application software for fleet management and carsharing in Java. He still lives in Hannover, Germany.

After having administrated Unix and Linux systems since the 1990s, I’ve rented my first real root server for myself only in 2016. While that machine worked and served things quite reliably, I felt it aging pretty fast. It was an Ubuntu 16.04 system, it had only 4 GB of RAM and a quite outdated Athlon64 processor.

Due to these parameters it was not possible to run virtualized setups - which I wanted to do to seperate services. E.g. that machine had a complete e-mail server setup which I intended to use for my private e-mail but still hesitated to activate as it was rather complex and "manually setup". I followed an instruction guide on the internet which even its author said is outdated only two years later.

There were other shortcomings like a less-than-optimal Let’s-encrypt setup. Let’s encrypt started almost at the same time as my server installation and there have been quite some optimisations since then.

All in all, my setup was aging, it was not sufficient for much more stuff on it and in the meantime, you got far more capable hardware for not so much more money.

So, in 2018 I decided to start "from scratch" with the system design. The first and most important design decision was about IP connectivity. IPv4 had run out of address space since more than five years then. IPv6 has been available for a decade or so. I wanted to do it in a modern way:

Implementing this scheme initially took me about a week. I wrote all steps down and published a series of blog articles in my personal blog. As systems evolve and interest remained, I continued to update the articles. Eventually, I came to the conclusion that this scheme of updating was not flexible enough. So, I decided to rewrite the articles (and some unpublished articles with further explanations) into an Asciidoc document and publish it on github. As usual, this project became a bit bigger as I expected and after integrating and updating all information I suddenly had a PDF document of 80 pages - which, as you see, has still grown since then.

When reworking this guide for Ubuntu 22.04 in December 2022, I decided to build an automated process for the basic steps of setting up the physical host. The result is install-kvm-host.sh which is now distributed together with this guide. This surely makes my life easier as I do not have to perform all the steps in this guide manually (and therefore prone to errors). But it should also help everyone who wants to setup a server this way.

Have fun reading this guide! I hope it helps you setting up your own, modern, IPv6-first setup.

If you rent the Hetzner server, order it with the "rescue system" booted. That gives the most control over how the system is configured. I suggest that you access the server in this early stage by its IP address only. We’ll change the IPv6 address of the system later in the install process. If you want to have a DNS entry, use something interim to throw away later, e.g. <plannedname>-install.example.org.

As suggested above, I obtained my server in Hetzner’s "rescue system" which allows the OS installation through the installimage script. I wanted to work as much as possible with default components and configurations, so I decided for the Ubuntu 22.04 ^[1] install image offered by Hetzner.

Logging into the new server gives you a welcoming login screen somehow like this:

You might want to write down MAC address and IP addresses of the system. Note, however, that they are also included in the delivery e-mail sent by Hetzner Online when the server is ready. You also see the addresses in the Hetzner Online server configuration interface or can request them with the ip -a command in the rescue system.

The system has two harddisks. I use them as a software RAID 1 as offered by the install script. This allows for at least some desaster recovery in case of a disk failure. And for systems like this, I do not install any partitions at all (apart from the Hetzner-suggested swap and /boot partition). The KVM disks will go to qcow2 files which are just put into the host’s file system. Modern file systems fortunately do not have any problems with 200+ GB files and this way, all the virtual guest harddisks are also covered by RAID.

Hetzner's installimage asks in a dialog for the image to use. This guide is applicable for the Ubuntu images, preferrably the 22.04 version, but 20.04 works also. 18.04 would be ok, too, but this version will be outdated in April 2023 ^[2] - don't use it for new installations any more.

The image installation process is controlled by a configuration file. Its (striped-down) version for the system I work on reads like this:

Installing the system this way brings a fresh and rather small Ubuntu system on the disk. Note that ssh will complain massively about the changed host key of the system, but that is ok. You’re now booting the installed system which has another host key than the rescue system you used before.

After having booted into it, I had some hours of remarkably degraded performance as the RAID 1 had to initialize the disk duplication completely. Be aware of this, your server will become faster once this is over. Use cat /proc/mdstat to see what’s going on on your harddisks.

If you install an e-mail server (or have some external mail service you want to use for system e-mails), you should enable alarming messages if the RAID degrades due to diskfailure. A RAID only protects against hardware failures if actually failed hardware is replaced quick enough.

This guide features the script install-kvm-host.sh which performs all steps following in this section on a Hetzner root server with a freshly installed Ubuntu 22.04. You may now fast forward to the description of install-kvm-host.sh to learn how to use it.

If any problems arise, you can go back to the following chapters to sort things out as the script really only performs the actions described here in the next sections.

If you want to do things on your own, if you work with another installation than Ubuntu 22.04 or if you are not on a Hetzner root server, read on to perform the setup steps manually.

You should check that the system supports virtualisation at all. Issue

and verify that the result is greater then 0. Then, apply

and check that the result is

If not, the BIOS settings of the system must be corrected. Contact the hosting provider to sort that out.

One thing which is totally independent from IPv6 and KVM is the /tmp directory. It contains temporary files. I like to put it into a ramdisk. Add one line to /etc/fstab and replace /tmp with the following commands:

This setup allows /tmp to grow up to 2 GB which is ok if the system has more than, say, 30 GB of memory. You can, of course, allow more or less. Note that the memory is only occupied if /tmp really stores that much data. An empty /tmp does not block any memory!

The mkdir creates /tmp without any special access rights. Fortunately, declaring the file system to be tmpfs in /etc/fstab above makes the access rights 1777 (or rwxrwxrwt) - which is exactly what we need for /tmp.

You should reboot the system after this change. Chances are that wiping /tmp this way confuses processes.

One tiny piece in the puzzle is the timezone of the just-installed machine. At least the Hetzner Online installation scheme leaves the server with UTC as timezone. If you want to have it in the local timezone, change it via

You get the available timezones with timedatectl list-timezone. For Germany, the command is timedatectl set-timezone "Europe/Berlin".

We do now have a freshly installed system. Unfortunately, it is not quite ready to serve as a KVM host. For this, we first have to configure a network bridge on the system.

I must say that I felt rather uncomfortable with Hetzner’s IPv6 approach in the beginning. Having only one /64 IPv6 network disallows a routed setup. Due to the way how IPv6 SLAAC address recovery works, you cannot split this network sensibly into smaller ones. I really suggest reading Why Allocating a /64 is Not Wasteful and Necessary and especially The Logic of Bad IPv6 Address Management to find out how the semantic of the IPv6 address space differs from IPv4. If you have a hoster who gives you a ::/56 or even ::/48 network, you can surely manage your addresses differently. Most probably, you will go with a routed setup.

However, since my start on the IPv6 road, I learned that Hetzner’s approach is not that wrong. They use the link local fe80:: address range for gateway definitions and this a totally valid approach.

We have to use what we get. First, enable IPv6 forwarding globally by issuing

Also enable this setting in /etc/sysctl.conf to make it permanent.

Now use ip a to get device name and MAC address of the physical network card of the system:

Your network device’s name may differ. It can be something like enpXsY as in this example or enoX. On all modern Linux distributions, it will begin with en, however…​

Here the common track for all systems ends. In the Linux world, multiple network configuration setups have evolved over time. The most common ones are:

Our virtual machines will have their own MAC addresses. Otherwise, the IPv6 auto configuration would not work. Unfortunately, these MAC addresses will also leak through the bridge into Hetzner's network and that might lead to trouble as the provider does only accept the actual assigned MAC address of the main server as valid.

To prevent such problems perform MAC address rewriting using the ebtables command. You might need to install it using apt install ebtables first. Then use:

I've added this to /etc/rc.local. On a default installation of Ubuntu 22.04 (or 20.04 - or 18.04), this file does not exist. If you create it, make it look like this:

Replace the address in the example with your actual physical MAC address! Also, make the file executable with chmod +x /etc/rc.local.

"The internet" claims that you need to add other files to systemd for /etc/rc.local being evaluated in Ubuntu. At least for me this was not needed, it "just worked". Check whether the rule has been added:

Reboot the systems once more to check if the rule survives a reboot.

You might think about changing the IPv6 address of the physical host. Hetzner Online configures them always having 0:0:0:2 as IPv6 address host part. While there is nothing wrong with that, giving the host a random address makes the whole installation a bit less vulnerable to brute-force attacks.

Fortunately, changing the address is really simple. In the Netplan-based setup, it is in /etc/netplan/01-netcfg.yaml. Look for the addresses of the br0 device:

Change it’s host part (the lower 64 bits) to more or less whatever you like

If you work with systemd-networkd, the network configuration is in /etc/systemd/network/21-br0-conf.network if you followed this guide:

Change it to

You can also add and not replace the additional address. Then, your server can be accessed through both addresses. While it is absolutely no problem to have multiple IPv6 addresses on the same device, it can make configuration of services more difficult as the correct address for outgoing messages has to be selected correctly. I would suggest not to do this. Stay with one IPv6 address.

Use netplan try or systemctl restart systemd-networkd to apply the new settings. Note that if you are connected via IPv6, your connection will be interrupted and you have to reconnect. If you are connected via IPv4 (e.g. by ssh <IPv4-address> or ssh -4 <hostname>), your connection should survive. systemd-networkd, however, might need several seconds to sort everything out.

If everything works, add a reboot. In theory, restarting the network configuration should be sufficient, but at least back in the days of Ubuntu 18.04 and earlier my system sometimes behaved strangely after this change.

ssh to the new address should now work. If it doesn’t and your are locked out, again use the rescue system to sort it out.

Now is the time to add the physical host to the DNS:

At this stage, lean back for a moment! The difficult part is done. You have the network setup of your KVM host up and running. The rest is much easier and will not potentially kill network access to system. Also, the stuff coming now is much less provider-specific. While the initial network setup might work considerably different with another hosting provider, chances are good that the following steps are the same regardless of where you have placed your host.

As I wrote, our virtual machines shall have IPv6-only internet access. That implies that they cannot access systems which are IPv4-only. Unfortunately, even in 2022 there are quite popular sites like github.com which do not have any IPv6 connectivity at all. To make such systems accessible from the guest systems, we setup a NAT64 service which performs a network address translation for exactly this case.

I decided to go with the "Tayga" server. It’s scope is limited to exactly perform NAT64. This makes it necessary to add further services to make all this really useable but it also minimizes configuration complexity.

In the last chapter, we have assumed that the IPv6-only system maps IPv4-only targets on a makeshift IPv6 address. The question remains how it is tricked into doing this. We solve this problem now.

With NAT64 and DNS64 in place, we’re almost ready to serve virtual machines on the host. The last missing bit is the network configuration.

Of course, you could configure your virtual hosts' network manually. However, IPv6 offers very nice auto-configuration mechanisms - and they are not difficult to install. The key component is the "router advertisement daemon". It’s more or less the IPv6-version of the notorious DHCP service used in IPv4 setups to centralize the IP address management.

For this service, we use the radvd router advertisement daemon on the bridge device so that our virtual machines get their network setup automatically by reading IPv6 router advertisements. Install radvd and also radvdump for testing through the usual Debian/Ubuntu apt install radvd radvdump.

Then, create the configuration file /etc/radvd.conf. It should contain the following definitions:

The route section advertises that this system routes the 64:ff9b:: network. Only with this definition the virtual servers know where to send the packets for the emulated IPv6 addresses for the IPv4-only servers to.

IPv6 route advertisement is prepared for dynamically changing routes. In our setup, however, all routes are static. Therefore, prefix and route advertisements are announced with "infinite" lifetime.

A radvd configuration must always be read as advertisement of the machine serving it. So, you do not write something like "service X is on machine Y" but "This machine offers X".

Having this in mind, the configuration advertises all three network settings needed by the virtual machines:

Start radvd and make it a permanent service (coming up automatically after reboot) using

If you start radvdump soon after starting radvd, you will see the announcements sent by radvd in irregular intervals. It should contain the network router, the DNS server and the NAT64 route. Note that radvd turns to rather long intervals between the advertisements after some time if noone is listening.

If you ever change the configuration, restart radvd and check its output with radvdump. It should contain both the DNS server and the NAT64 route.

You have now prepared everything for the IPv6-only virtual machines to come: They get their network configuration through the centrally administrated radvd. The advertised setup includes a name server with DNS64 an a NAT64 route to access IPv4-only systems.

For Ubuntu, I followed the first steps of this guide. On Ubuntu 22.04 the installation command is

On Ubuntu 20.04 or 18.04 the list of packages is slightly different

This will install a rather large number of new packages on your host. Finally, it will be capable to serve virtual machines.

Next step is to load the vhost_net module into the kernel and make it available permanently. This increases the efficiency of networking for the virtual machines as more work can be done in kernel context and data can be copied less often within the system. Issue two commands:

The libvirtd daemon should already be up and running at this point. If this is for any reason not the case, start and enable it with the usual systemctl commands or whatever the init system of your host server requires to do this.

To simplify installation and administration of your virtual machines, you should add a "normal" user to the system and allow that user to administrate the virtual machines.

You should perform a final reboot after these steps to be sure that everything works together correctly and comes up again after a reboot.

Well, that’s it! Our system can get its first virtual machine!

Currently, install-kvm-host.sh can only be used if

In this case, generally perform the following steps:

The script will perform all the steps described in the previous sections. The steps are grouped into three stages:

In normal operation mode, install-kvm-host.sh will reboot the server between these stages to ensure a stable, predictable configuration.

On the system I developed and tested the script, the stages needed the following time:

The main part of the time is actually spent downloading the needed packages. The configuring steps only need some seconds in each stage.

My server also needed up to 7 minutes for each reboot. The reason for these delays is unknown. Hopefully this is special to that hardware.

As the system reboots in between, the connection to the console is lost. The script logs all console output also to /var/log/install-kvm.log so that one knows what actually happened and whether any errors occurred.

Currently, discovering the end of the script’s operation is done by monitoring the server. If it has rebooted three times, the script has finished. If it has been successful, there is neither a /var/lib/install-kvm/config nor a /var/lib/install-kvm/stage file. In any case, /var/log/install-kvm.log contains the output of all (executed) script stages and ends either in a success message or with an error message describing what went wrong.

install-kvm-host.sh has three operation modes:

Currently, install-kvm-host.sh knows about two optional operations:

install-kvm-host.sh is a lengthy Bash script. It has some nice features which are described a bit more in depth here

Installing a virtual machine might sound difficult to you if you have never done this before. In fact, it is not. After all, it is even simpler than the remote installation on a hosted system once you get used to it. On the physical machine, you are dependent on what the hosting provider offers as installation procedures. KVM offers you more or less a complete virtualized graphical console which allows to act just as you were sitting in front the (virtual) computer’s monitor. This way, you can install whatever you like.

Furthermore, if you make a configuration mistake on the physical host, you might end with a broken machine. If this happens in a virtual machine, you have several ways to solve the problem: You can connect to the console and log in directly without network access. If the machine does not boot any more, you can even mount the virtual hard disk into the physical machine and try to fix. And if the machine is for any reason broken beyond repair, you can just throw it away and start over with a fresh installation.

I suggest to start with a throw-away virtual machine. It will not contain any "real" services but only show any remaining problems in the setup. Furthermore, it allows to test and learn the whole process of installing a virtual machine in the setup:

From hereon, things become rather standard. We’re now in the process of installing a guest system on a KVM host. My best practices are these:

The one interesting point is the network configuration at the very end of the definition process. Here, enter the "Network selection" before creating the machine. Select "Bridge device" as network type and give the name of bridge which is br0 in our setup:

If you are ready, press "Create" and summon your first virtual system.

To work with the long IPv6 addresses conveniently, DNS is almost mandatory. You should enter the virtual machine now into the domain.

The SLAAC mechanism derives the IP address from the MAC address of the virtual machine. So, it will be static even though it has nowhere been configured explicitly.

If you want your virtual machine to be started automatically if the physical host starts up, you have to set the corresponding flag in the KVM configuration. In virt-manager, you find it in the "Boot options" for the virtual machine.

The first advice for this topic: Don’t do it!

Your virtual machine is fully connected to the internet any can be reached from anywhere and for any service. The only restriction is that a connecting party must use IPv6. Most systems, however, are connected by IPv6 these days.

Even if the virtual machine hosts a service which must also be accessible by IPv4-only clients, a direct IPv4 connection for the virtual machine is not mandatory. Especially for HTTP-based services, we will configure a reverse proxying scheme on the physical host which allows transparent connections to IPv6-only web servers on the virtual machines for IPv4-only clients. So, only configure a direct IPv4 connection into a virtual machine if it runs a service which requires it.

In the provider-based scenarios, the provider has to assign an additional IPv4 address to the physical host. Then, the physical host must be configured in a way that it passes the IPv4 address to the virtual machine. Finally, the operating system on the virtual machine must handle these connections. Do like this:

Obtain an additional IPv4 address for your server from your hoster. If you already have a network of them assigned to your server, you can use one of those, of course.

On some systems, namely Ubuntu 20.04, systemd-networkd has a problem detecting that all networks came up on boot of the virtual machine. Typical symptoms are

If you face such symptoms, check if systemd-networkd is the reason. Just run as root on the command line:

If this command issues the mentioned Event loop failed error message on the command line after 3 seconds, you have this problem.

The actual reason for this problem is a bit unclear. There seem to be some glitches in the way systemd-networkd detects the network availability in virtual machines under certain circumstances.

However, as the virtual machines do not have more than one network anyway it is enough to wait until any network of the machine became available. Issue

It should return immediately without error message.

To make this workaround permanent, change the configuration of the systemd-networkd-wait-online service. There is a default way of doing this in systemd:

You’re done! Rebooting the virtual machine should now be a question of seconds instead of minutes.

In this section, I give some advises for the Ubuntu 22.04 installer. Note that Ubuntu has changed the installer completely compared to Ubuntu 20.04 so the menu is much different.

If you install the system from the "Create virtual machine" screen, the install ISO image will be removed from the boot device sequence of the virtual machine automatically. However, if you installed the system into an already existing virtual machine, you have to do that yourself. This change will only be applied if you turn off the virtual machine completely and restart it from scratch (e.g. in virt-manager). Rebooting the operating system within the virtual machine is not enough!

Subiquity writes an extremly stripped down Netplan configuration into /etc/netplan/00-installer-config.yaml. It contains no information about IPv6 autoconfiguration; it seems that SLAAC is always performed, even if not explicitly mentioned. If you left the default network settings as they were, however, there is a line enabling DHCP for IPv4. I suggest to disable this by setting it to false. Note that you should not remove all lines completely as there must be at least one entry for an active network device in the Netplan configuration.

NixOS follows a unique declarative configuration style. The whole system is configured by a static description which is rolled out by "nix", a kind of preprocessor and package manager. NixOS offers capabilities as atomic updates and per-package dependency chains, i.e. it allows to have multiple versions of libraries and even programms installed in parallel.

For Ubuntu 20.04, I advise these steps:

As this is Ubuntu 20.04, this machine uses netplan for network configuration. It has a very simple definition file in /etc/netplan/01-netcfg.yaml:

Netplan summarizes router advertisement in the "dhcp6" statement. You do not need to change anything here.

Note that after (re-)booting the virtual machine, it may take some seconds until it has configured its network interface. Once it has done so, everything should work without problems.

Check if you have the network setup delay problem after reboots and fix them as described in the referenced section.

Ubuntu 18.04 is installed exactly the same way as Ubuntu 20.04.

Windows "just works" when installed in an IPv6-only virtual machine. It takes all the advertisements from radvd and configures its network appropriately. The result is a Windows which has no IPv4 network aprt from the auto-configured one.

Lets recap shortly how SSL works:

The mentioned 3rd party is the certification authority (CA). To issue a valid certificate, it must answer two questions positively:

Let’s encrypt, however, uses a completely automatable workflow which is based on so-called challenges. The protocol and scheme are called "ACME" and work this way:

There are different types of certificates. First, you can certify a key for only one special server name, e.g. "www.example.org"; then, only a service which uses exactly this name can use the certificate. This is the type of certificate you read most about when it comes to Let’s Encrypt. Its challenge is rather simple:

The very nice thing about this scheme is that the Let’s-Encrypt server, the Let’s-Encrypt client certbot and your web server can handle out this all on their own. certbot knows how to configure a local Apache so that it serves the file which the Let’s Encrypt server wants to see. There are even web servers like Caddy which have this scheme already built-in.

However, for the mixed IPv4/IPv6 setups, this scheme has some shortcomings. It can only be performed on the virtual machine, as only there the web server can change the contents it serves in accordance with the Let’s encrypt certificate request tool. While this would work consistently as Let’s encrypt explains on https://letsencrypt.org/docs/ipv6-support/, the key has to be copied to the IPv4 proxy server. Otherwise, encrypted IPv4 connections are not possible. If for any reason the IPv6 web server is unresponsive and Let’s encrypt falls back to IPv4, the challenge fails and the certificate is not issued.

All in all, I decided against this scheme for all my domains.

In my setup, I use a certificate for any name below example.org. Such a wildcard certificate is expressed as *.example.org and allows to use the certified key for everything below the domain name, e.g. www.example.org, smtp.example.org, imap.example.org and anything else.

For such wildcard certificates, the HTTP-based challenge is not sufficient. You have to prove that you not only control one name in the domain, but the complete domain. Fortunately, there is a way to automate even this: The DNS entries of the domain.

Obtaining a certificate for *.example.org works with the following dialog:

This works perfectly. There is one tiny problem: The DNS entries have to be changed. And there is no general scheme how to automate this with the usual DNS server system. Many providers do only offer a web-based interface for editing or have a proprietary interface for changes to the DNS entries. And such interfaces tend to give very much power to the caller.

Thankfully, there is a solution even for this problem: A special tiny DNS server which is tightly bound to the whole challenge process and can be programmed to return the required challenge. Such a server exists, it is the "ACME-DNS" server on https://github.com/joohoi/acme-dns.

But this server runs on its own name which might even be in a totally different domain, e.g. acme-dns.beispiel.de.

And here comes the final piece: The domain example.org gets a static CNAME entry pointing acme-challenge.example.com to a DNS TXT record in acme-dns.beispiel.de. And _that entry is the required TXT record Let’s Encrypt’s certification server checks for the certificate validation.

The complete scheme with ACME, ACME-DNS and all the routing steps becomes rather elaborate:

This scheme finally allows to automatically obtain and manage Let’s-Encrypt-certified SSL wild card keys. And full automatism is important for the whole Let’s Encrypt scheme as the issued certificates only have rather short validity time ranges: They are only valid for 90 days and can be renewed from day 60 of their validity on. So, each certificate is renewed four to six times per year!

Especially when the number of certificates becomes larger, a manual process is really, really tedious.

Time to bring things together. We setup the following scheme:

In my original installation, I chose the physical host as the hub for this scheme. While there is nothing wrong with that technically, it contradicts the rule not to run any unnecessary services at the physical host.

However you do it, first thing you need is the certificate robot certbot. In all Ubuntu versions covered here, the included one is sufficient, so installation is as simple as apt install certbot.

For the ACME-DNS part, you need an ACME-DNS server. For the moment, I decided to go with the public service offered by acme-dns.io. As its administrators say, this has security implications: You use an external service in the signing process of your SSL certificates. Wrong-doers could tamper your domain or get valid certificates for your domains. However, as we see, you’ll restrict access to your data to your own server, you are the only one who can change the crucial CNAME entry in your domain and you have an unpredictable account name on the ACME-DNS server. As long as you trust the server provider, this all seems secure enough to me.

It is absolutely possible and even encouraged by their makers to setup your own ACME-DNS instance, and as the server is a more or less self-contained Go binary, it seems to be not overly complicated. However, this has to be elaborated further.

Finally, you need the ACME-DNS client. It is a short Python program. Download it into /etc/acme-dns from its source on github:

Load the file into and editor end edit the configuration at top of the file:

Now you can generate key and certificate for your first domain. Run certbot on a domain to be secured with SSL once manually:

Executing this call halts just before Certbot checks the certificate. Instead, it prints the required CNAME for the domain. So, now you have to edit your domain data and add this CNAME.

After the entry has been added and spread through the DNS, let certbot continue.

Only at this first certification process, manual interaction is needed. After that, certbot can, together with acme-dns-auth.py, handle the whole process automatically. You can see the setup in /etc/letsencrypt/renewal/example.com.conf.

For the process of certification that means: Put a call to certbot renew into the weekly cronjobs on the machine and forget about it.

Certbot puts the currently valid keys certificates for a domain into a directory /etc/letsencrypt/live/<domain>. Just use the files in that directory for Apache, Dovecot, Postfix and whatever.

My suggestion is to run Certbot in a centralized way on one machine (in my current setup: the physical host) and copy the complete directory with certbot’s keys and certificates onto all machines which need one of the certificates.

Note that you need the certificates for all your web servers not only on the virtual machine where the web site runs, but also on the physical host which is responsible for the incoming IPv4 connections to the site. It helps to prevent errors if you just copy the complete data set onto all machines at the same place.

So, just copy the /etc/letsencrypt directory to all virtual machines, do not install certbot there.

From all services described in this guide, the Let’s Encrypt setup is the least consolidated one. There are some elements missing to make it really polished:

Taking all this into consideration, this section of this guide is the clearest candidate for a larger rewrite.

So, here we are. Start with bringing up a fresh virtual machine as described above. I gave my e-mail server a virtual harddisk of 250 GB - of which 60 GB are currently used. It has 6 GB of main memory. This is rather a lot, but in my setup, there are 64 GB of physical memory in the host. It is no problem…​

Perform all steps so that you have your e-mail server-to-be accessible from outside via IPv6 and IPv4 ssh connections with appropriate DNS entries in your domain.

After having the server prepared on network and system level, it is time to actually setup the mail system. And here I must admit, that I have not developed my own scheme. Instead, I followed https://thomas-leister.de/en/mailserver-debian-stretch/. While it uses Debian stretch as base, I also had no problem following it on Ubuntu 18.04.

Thomas describes a contemporary e-mail server setup. The server allows for incoming and outgoing traffic. It is setup according to all anti-spam requirements with DKIP, DMARC and SPF entries in DNS. It offers SMTP and IMAP access for clients. It is a multi-user multi-domain setup. I run this since late 2018 without any problems.

Using Thomas' description is very straight forward. However, I’d like to add some notes.

First, it helped me tremendously to understand that you should use a totally unrelated domain for the mail server - especially, if you plan to serve multiple domains on it. Let’s assume you want to serve example.net and example.com e-mail on your server. My advise to setup this:

Setting things up like this makes the further DNS setup rather simple: All your mail clients connect to the dedicated mail server domain, in the example mx0.beispiel.de, also those for example.net, example.com or whatever. This way, you only need SSL certificates for the mail domain.

Two things are absolutely crucial regarding the name service setup of the mail server:

As I already explained, the mail domains which are hosted on the system can have totally different names! There are also no PTR records needed. Each mail domain must match the following conditions:

I use this setup since autumn of 2018 without any problems. I’ve also added a number of domains to my server. This takes only minutes, works without problems and covers all kinds of setups like local delivery, delivery to multiple addresses, forwarding, catch-all addresses. It is a really fine thing.

You might miss a webmail interface. I have decided not to add that to the mail server, but to another virtual web servers. Setup of those is covered in the next chapter.

For this scheme to work, DNS entries must be setup in a special way: For a web site www.example.org, you define an A record which points to the physical host but an AAAA record which points to the virtual machine.

After all, this is the main idea behind everything which is following now.

For the following examples, we assume that the following DNS entries exist:

Install apache2. Then, enable rewrite using a2enmod ssl rewrite. This is needed for encryption and the mandatory redirect from http to https.

Install apache on the physical host, too. You have to enable the proxy modules additionally to SSL and rewrite using a2enmod ssl rewrite proxy proxy_http.

Create the site configuration, also in /etc/apache2/sites-available/testsite.conf:

Enable it with a2ensite www.example.org. Now, your website is also available via IPv4.

Note that the Apache on the physical host resolves its proxy target simply as www.example.org. This works as by specification IPv6 name resolution always superceeds IPv4 name resolution. This way, the physical host actually forwards the incoming request to the real server on the virtual machine.

Note that we define the IPv4 redirection server directly on the physical host and not as forwarder to the HTTP-definition in the virtual machine. This way, proxied requests are reduced.

To restrict access to port 3000, we need a firewall. Ubuntu comes with ufw which does the job. Install it with apt install ufw. Then perform the following commands for the basic setup:

If you have other services on the virtual machine, add their respective ports with more allow commands. You can get the list of installed application packages with ufw app list. Note that you do not need a local Postfix instance for cryptpad.

ufw stores the actual firewall rules in files below /etc/ufw. The app rules defined above get stored in /etc/ufw/user6.rules. You should not temper that file, stay with ufw commands for configuration. If everything is in place, change /etc/ufw/ufw.conf and set

To check that everything works as expected, perform ip6tables -L which should be empty now. Start ufw with systemctl restart ufw and run ip6tables -L again. Now you should see a rather lengthy list of rule chains and rules, among them the rules regarding port 3000 you gave above.

Test that you can reach the Apache server on the virtual machine.

You should now add the DNS name of your cryptpad instance. Remember what we said about system names and service names: The virtual machine Cryptpad will run on is not accessible via IPv4 directly. Therefore, you need a proxy on the IPv4-accessible physical host of your installation. As a result, the DNS entries for accessing your Cryptpad instance will point to different servers in their A and AAAA records. To avoid confusion, use the Cryptpad service entries only for accessing the service and use the name of the virtual machine for maintenance.

This said, add the appropriate entries to your DNS records. We will assume cryptpad.example.com as name for the Cryptpad service in this guide.

Cryptpad is open-source software. Their producers offer storage space on their own cryptpad servers as business model. Therefore, they are not overly eager to promote independent installations. Nevertheless, it is no problem to run and maintain a self-hosted installation of the software as long as you have some idea about what you are doing.

Start with creating a user "cryptpad" with adduser --disabled-password --gecos "Cryptpad service" cryptpad. Add your key in its .ssh/authorized_keys file so that you can log into the account.

As user cryptpad, you install some software packages needed by Cryptpad:

This installs the Node package manager npm and sets paths correctly. Check it with

Now you are ready to actually install Cryptpad.

Stay as user cryptpad and start by obtaining the software:

This installs Cryptpad right from its Github repositories. It’s the default way of installing an independent Cryptpad instance. Installing this way has the big advantage that updating to a newer Cryptpad version is a simple git pull, followed by the instructions in the version announcement.

Now you perform the basic setup:

These are also the routine steps after an update. Note that especially the npm install step seems to download "half the internet". This is expected behaviour. Cryptpad comes from the Javascript/Node.js sphere and those folks love to use a plethora of library packages which themselves use another plethora of library packages. Fortunately, subsequent updates will only touch a subset of these libraries…

After installation comes configuration:

Now you edit config.js. It’s a JSON file and there are three important changes to be performed:

We configure cryptpad itself in a way that it uses it’s domain name in the _domain variable. httpAddress is the actual address cryptpad starts its own HTTP server on. To be sure that this happens on the correct interface, we use the actual IPv6 address here.

After this step, Cryptpad is configured completely but not yet started. We come back to this in a moment.

As a final step, you should remove the $HOME/cryptpad/customize subdirectory if you do not really need it. It will not be updated during updates and might carry outdated information after updates.

Usually, Cryptpad is a service which runs permanently. Therefore, it should be started on system startup. For systemd-controlled servers as any modern Debian or Ubuntu installation, add a systemd service unit file:

Symlink this file into /etc/systemd/system:

Now you can start the cryptpad service:

The cryptpad server should now be reachable on the virtual machine and on the physical host on its port 3000. Test it with curl http://cryptpad.example.com:3000/ on both systems. From any other computer, the service must not be reachable due to the firewall blocking the access. Test this, too!

To make Cryptpad accessible from the outside, we configure an Apache proxy. For Cryptpad, it needs to proxy websockets, so (as root) run the command a2enmod proxy_wstunnel.

Then, the Apache configuration is rather straight forward:

After that, activate the virtual host:

If everything comes up without errors, you can access your Cryptpad from any IPv6-connected computer. Check that loading a pad actually works, otherwise there is a problem with the forwarding rule for the websocket.

The Cryptpad instance is not yet accessible from IPv4 clients. For this, you need another Apache proxy on the physical host. The very nice thing here is that it can be configured exactly as its compaignon on the virtual machine! So, on the physical host as root do this:

Now, cryptpad can also be reached from any IPv4-only hosts.

Note that on the physical host, you forward to port 3000 on the virtual machine. This is the reason why the port must be reachable from the physical host. Port 3000 on the physical host is totally unaffected from all of this and in fact, you could just install another, independent service there without breaking your Cryptpad on the virtual machine.

The main takeaway from this installation procedure is: If you have a service on a virtual machine for which you configure an Apache (or Nginx or any other HTTP server) as reverse proxy, chances are good that you get IPv4 connectivity by adding another reverse proxy to the physical host (or any other IPv4-connected machine) with exactly the same reverse proxy configuration. A future version of this guide should make this clearer.

In Netplan, this configuration is very simple to archieve:

Note that eth1 must be mentioned somewhere in the configuration, otherwise it will not be brought up by the kernel. Then, the route is simply added. netplan try or netplan apply activates it and of course it will be setup again after a reboot.

The same way, you configure phyB, but with the network and link-local via address of phyA:

Configuring the additional static routes on all systems explicitly can become rather tedious. Fortunately, if you have radvd running on the system - as we have on our physical hosts - it can also do this job. The idea is that each system which is connected to the local switch just announces itself as gateway to its network. On phyA, the according definition looks like this:

With AdvSendAdvert we enable information advertisment generally. AdvDefaultLifetime 0 declares that this system is not a default router but handles only traffic for the targets explicitly mentioned in the routes. Finally, route declares the route to the network of this system. AdvRouteLifetime infinity declares this route to be valid forever.

Equipping all servers connected to the local switch will make all servers connected to all other servers through their respective eth1 links. The routing table of phyA will look like this:

IPv6 has some routing concepts which are rather different from how IPv4 works. Using the link-local IPv6 addresses of the interfaces has significant advantages over the usual "private address" approach of IPv4 in such situations: On IP level, it is totally transparent over which interface the packets are sent. phyB is always phyB, regardless whether it is connected from phyA or from anywhere else. Only phyA, however, will use the local switch for connecting, while everyone else will go over the global infrastructure. And on phyA, you do not have to think about routing issues. You just connect to phyB and in the background, routing is done through the appropriate interface.

Shorewall is a quite sophisticated system to manage the complete network routing rules on a host. It replaces all rules in the routing tables. This can render services unusable if you do not configure them in Shorewall’s own rules.

Shorewall separates IPv4 and IPv6 strictly from each other in all configuration steps. While this makes sense from the protocol point of view, you have to always keep both protocols in mind and might repeat some configuration steps for both protocols.

Following Shorewall’s structure, we look at IPv6 and IPv4 configuration separately.

Of course, you may install Shorewall (or any other firewall system) also on the virtual machines. You must even do so if you want to restrict access to services as these connections generally do not go through the physical machine’s network stack.

If your virtual machines are IPv6-only machines (as this guide recommends), you only have to care about IPv6 traffic. The machine will never see any IPv4 packets from the outside.

If your virtual machine has direct IPv4 connectivity with an official IP address, you have to take care for it in the firewall. For both protocols you configure the firewall just as if the machine was a stand-alone system. Direct incoming traffic is in both cases unaffected of the physical machine.

Note that even on virtual hosts with direct IPv4 connectivity, outgoing connections to IPv4 targets might still be passed through the DNS64/NAT64 layer so that the target machine will see the connection as opened from the physical host. The e-mail setup notes describe how to change that (just do not use the DNS server of the physical host).

The Shorewall rules recommended above enable ssh unconditionally from everywhere. While this usually has no security implications, log files clobbered with warnings become quite annoying. A simple and quite effective solution is the small "sshguard" package available for many distributions.

sshguard scans the logs for failed ssh logins. If there were too many attempts from one source, it creates a temporary rule in the network stack dropping all network traffic from that source.

sshguard can be installed alongside Shorewall without problems. If you use it, install it on the physical host and each virtual machine as each installation only knows about its own system.

If you have trusted networks which connect to the machine regulary, consider adding these networks to sshguard’s whitelist in /etc/sshguard/whitelist. Then, sshguard will not block these addresses even if some failed logins are logged. You can whitelist IPv4 and IPv6 addresses and address ranges in any order and combination.

The Java Runtime Environment supports IPv6 for a long time now. Actually, they started supporting it in a time when IPv6 implementations were far from mature - being it in Java itself or in the operating system. To be able to supercede the IPv6-preference, the JRE got a property java.net.preferIPv4Stack. If this is set to true, Java always prefers IPv4 over IPv6 if both protocols were available for a connection.

In a NAT64/DNS64 environment, this results in severe problems: For an IPv4-only host and even for hosts reachable via both IP stacks, the DNS64 server delivers both the IPv4 and the (potential NAT64-routing) IPv6 address. Preferring IPv4 over IPv6 in such a case means trying to connect over the (unreachable) IPv4 path even though a working IPv6 route would be available. Connection errors and timeouts are the result.

If your Java program has problems in the IPv6-only environment, check if this option is still set somewhere, e.g. as a -D option in the command line or in some options. Hopefully it is not compiled in. Remove it and connections should go through again, either via genuine IPv6 or via the NAT64 relay.