How to run Wireguard VPN in balenaOS

Execution time: 1hr - 2hr

Difficulty: Medium

Cost: Low

This guide will teach you how to run Wireguard VPN using balenaOS. Wireguard is a simple, modern, and fast VPN solution that suits balenaOS and balenaCloud use cases well.

We created this guide to address some user questions that we’ve received a few times on our forums. As with any solution at balena, it’s continuously improving to help our users with the various ways they put balena to work. Chime in via our Forums if you give it a try or have any suggestions or improvements.

Implementing Wireguard VPN

This build guide will go over the following steps:

Building the kernel module and tools
Loading the module & configuring the tunnel
Running the service

Stage 1: Building the kernel module and tools

Wireguard instructions on building from source are quite clear:

$ sudo apt-get install libelf-dev linux-headers-$(uname -r) build-essential pkg-config

Note: this is on a system running Debian/Ubuntu and the kernel headers for the running kernel are needed. We will be building on balena’s cloud builders, for the correct architecture of our target devices, and in this case, x86_64. We cannot use the command above verbatim since the builders do not use the same kernel version as our balenaOS target device. So, we need to pull the kernel header source ourselves.

An important part of running software in containers is that, in the words of Kelsey Hightower, “(we) should ship artifacts, not development environments,” and in the world of low-resource devices, this is increasingly important.

This means we want to use these build tools and not carry them over into our final image. We have a means to do this in the form of Docker Multistage builds.

Here is the start of our Dockerfile:

FROM balenalib/amd64-debian as builder

RUN install_packages curl build-essential libelf-dev libssl-dev pkg-config git flex bison bc python kmod

...

The important part here is the first line, where we define our first stage image and name it builder. This will let us reference the image that is built for this stage later, and copy files out of it into our final image.

The command install_packages is added to the image by balena to make adding packages simpler and neater in the Dockerfile.

Looking to the Wireguard documentation on building from source, we see that we should checkout the code from their Git repository. So let’s add that into our Dockerfile in the builder stage:

WORKDIR /usr/src/app

RUN git clone https://git.zx2c4.com/wireguard-linux-compat && \
git clone https://git.zx2c4.com/wireguard-tools

Now, this source code will need to be compiled against the target kernel source. So let’s bring that into our image too:

ENV VERSION '2.48.0+rev3.dev'
ENV BALENA_MACHINE_NAME 'intel-nuc'

RUN curl -L -o headers.tar.gz $(echo "https://files.balena-cloud.com/images/$BALENA_MACHINE_NAME/$VERSION/kernel_modules_headers.tar.gz" | sed -e 's/+/%2B/') && \
tar -xf headers.tar.gz

In this instance, I have made the balenaOS version and target device variables. This is because we can leverage the features of the balenaCloud builders to template this file and make it compatible with our chosen target device type. For now, I will leave this as an exercise for the reader.

Now we have the source files, it's time to do some compiling. First, we need to prepare the module source:

RUN ln -s /lib64/ld-linux-x86-64.so.2  /lib/ld-linux-x86-64.so.2 || true
RUN make -C kernel_modules_headers -j$(nproc) modules_prepare

Now the main event:

RUN make -C kernel_modules_headers M=$(pwd)/wireguard-linux-compat/src -j$(nproc)
RUN make -C $(pwd)/wireguard-tools/src -j$(nproc) && \
    mkdir -p $(pwd)/tools && \
    make -C $(pwd)/wireguard-tools/src DESTDIR=$(pwd)/tools install

After these steps we should have all the required Wireguard module and tooling compiled and linked to our target kernel.

Stage 2: Loading the module & configuring the tunnel

We want to continue with our Dockerfile in order to produce a smaller and more finely scoped image with just our binary artifacts and none of the development code or tools. To do this, we should add another FROM directive into the Dockerfile:

FROM balenalib/amd64-debian

This one isn’t named and will be the final image we define. It will be this image which is pushed to your balenaCloud release and onto your fleet of devices.

We want to bring over the kernel module and the tools, so let’s add a COPY directive:

WORKDIR /wireguard
COPY --from=builder /usr/src/app/wireguard-linux-compat/src/wireguard.ko .
COPY --from=builder /usr/src/app/tools /

Notice the --from= arguments to the COPY directives; this is informing the Docker build process to find these files in a previous named-stage from earlier in the build.

Files copied it is time to add some tools to allow us to actually load the module at runtime:

RUN install_packages kmod

At this point we have our image prepared, but we need to make it runnable. For this example, I will use an ENTRYPOINT script to load the required modules and a default CMD which will be used to configure the tunnel. The ENTRYPOINT script will look like this:

#!/bin/sh

modprobe udp_tunnel
modprobe ip6_udp_tunnel
insmod /wireguard/wireguard.ko || true

exec "$@"

… and for our CMD script we will just take the example config from Wireguard’s contrib, which sets up a tunnel to their demo/test server, and add on a command to the end to make the process idle:

#!/bin/bash
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (C) 2015-2020 Jason A. Donenfeld <[email protected]>. All Rights Reserved.

set -e
[[ $UID == 0 ]] || { echo "You must be root to run this."; exit 1; }
exec 3<>/dev/tcp/demo.wireguard.com/42912
privatekey="$(wg genkey)"
wg pubkey <<<"$privatekey" >&3
IFS=: read -r status server_pubkey server_port internal_ip <&3
[[ $status == OK ]]
ip link del dev wg0 2>/dev/null || true
ip link add dev wg0 type wireguard
wg set wg0 private-key <(echo "$privatekey") peer "$server_pubkey" allowed-ips 0.0.0.0/0 endpoint "demo.wireguard.com:$server_port" persistent-keepalive 25
ip address add "$internal_ip"/24 dev wg0
ip link set up dev wg0
if [ "$1" == "default-route" ]; then
host="$(wg show wg0 endpoints | sed -n 's/.\t(.):./\1/p')"
ip route add $(ip route get $host | sed '/ via [0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/{s/^(. via [0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}).*/\1/}' | head -n 1) 2>/dev/null || true
ip route add 0/1 dev wg0
ip route add 128/1 dev wg0
fi

exec balena-idle

These can then be added to the Dockerfile, to complete our service image:

COPY client.sh ./
COPY entrypoint.sh /entrypoint.sh

ENTRYPOINT [ "/entrypoint.sh" ]
CMD [ "/wireguard/client.sh" ]

Stage 3: Running the service

In order for the service the have the correct privileges to load the module, create the tunnel and use it you will need to define some extra directives in the docker-compose file:

services:
  wireguard:
    build: ./wireguard
    privileged: true
    network_mode: host
    labels:
      io.balena.features.kernel-modules: 1

The service container must run privileged, and it must also have access to the host’s kernel modules in order to load the dependencies in the ENTRYPOINT script. In this example we have also put the container into the host networking namespace. This means that the Wireguard tunnel will be created at the host level.

Try the implementation

We have looked at a few topics here, from building kernel modules against the balenaOS kernel source to running private VPN stacks on the device. The scope of this article is just to highlight the possibilities rather than to show specific implementations and as such you should adapt these scripts to suit your individual needs.

Try using Wireguard VPN and if you get stuck, have questions, or want to show off your success, reach out to us on the Forums. Good luck and happy coding.