Content

30 October 2020 / Last updated: 25 Jan 2021

Receive and store LoRa sensor data from TTN using a Raspberry Pi (part 2)

Execution time: 4hours +

Difficulty: Medium

Cost: High

Back in the first part of this series, we detailed the process of building a simple LoRa-connected weather station to monitor temperature, humidity, and pressure, and send the readings to The Things Stack (TTS). In this second part, we’re going to take a look at how you can capture that data, record it and produce a dashboard and graphs to visualize the data.

Note: 28th June 2021: The Things Network (TTN) is moving officially to The Things Stack (TTS). TTN is no longer available to create LoRaWAN gateways, so we’ve reflected this guide to represent those changes.

Introduction

When our weather station transmits data, the gateway receives the transmission and forwards it to TTS. If it isn’t able to be forwarded to some other system for storage, it’s discarded. Obviously, this is no use if you want to build up a history of weather data and start to maintain records. So, what do we do about it?

The answer is to use the ‘Integrations’ section within your TTS application to forward the data on. There are numerous cloud services available that will allow you to do this. A couple of examples are Ubidots and Datacake; both of these services are able to connect with TTS and receive your data.

However, if you wish to store an extensive history, you’ll have to pay a small amount for the privilege. This is fair, of course, as in that case we’re asking these services to hold our data indefinitely which depends on storage space on systems that have associated costs.

If you’re willing to set up your own device though, you can take responsibility for your own data, so let’s take a look at that; we’re going to set up a Raspberry Pi and configure TTS to send data to it via an HTTP integration.

We’re going to set up the Raspberry Pi using balenaBlocks; ready-to-use services that we can use to get up and running quickly with zero configuration. This is going to be a little different from our usual guides in that we haven’t built this project and hosted it on GitHub. Instead, we’re going to show you how to create it!

You’ll need to be familiar with code editors and the command line interface on your chosen operating system; this isn’t a super difficult process and is a great next step if you’ve already deployed some of our other projects in the past, but isn’t aimed at the complete beginner.

Hardware required

I’m going to be using a Raspberry Pi 3 to receive the data for this project and so I’ve listed these components below, but you could also use many other device types as what we’re doing isn’t Pi-specific. Additionally there are also cloud providers you can use that won’t require any hardware to receive the data; it depends if you want to have total control of your data, or not.

Raspberry Pi (3, or 4)
An SD card (we always recommend SanDisk Extreme Pro SD cards)
Power supply

Software required

The balena CLI tools
A code editor (e.g. VSCode)
A free balenaCloud account
An SD card flashing tool (e.g. Etcher)

App structure

In order to receive data from TTS, we need a listening HTTP server. If we set up a HTTP integration on TTS it will make a post request with the data from our sensor to the URL we specify.
First, we need something that listens and receives these requests. Next, we’ll need somewhere to store the data after we receive it, so we’ll need a database. Then, we’ll need some way to visualize the data and make it easy to read, identify trends, maintain record values, etc.

That’s three core functions that our device needs to perform in order to carry out this task. Fortunately, these are very common pieces of functionality that are present in a lot of different applications, and so there are balenaBlocks available for all three! Yaaaas!

Building a balena app with balenaBlocks

OK so let’s build something. You’re going to need to familiarize yourself with the balena basics here. If you’ve already built one of our other projects, you should know what to do. If this is your first balena project, I recommend taking a look at our getting started guides first.

Step 0: Install the prerequisites!

Before we get started, make sure you have all the items from the software required section installed and working. This means your code editor, the balena CLI tools and Etcher.

Step 1: Set up an application and device within balenaCloud

This is the same as any other balenaCloud project; create an account if you don’t have one already, create an application being sure to select the correct device type for the device you’re using (we’re using a Raspberry Pi 3 (not 64-bit)), and then add a device to that application. Flash the SD card with the downloaded image and boot the device.

For more information, take a look at the getting started guide and follow it up until adding a release. Stop there as we’ll make our own release here.

Add a new application and device to your balenaCloud account

When you’ve finished step 1, you should be at the point where you have an application set up with a device added within balenaCloud. You’ll notice that device is not doing anything yet (as indicated by ‘Factory build’ being shown under ‘Current release’. Onward!

Step 2: Create a project folder and compose file

Create a folder on your computer, create a new file called docker-compose.yml, and open this in your code editor (I’m using VS Code).

Edit the empty file so that it includes the following code:

version: '2.1'
volumes:
    database:
    dashboard:
services:

The version is the version of compose file we’re using, and the volumes entry sets up a data volume called data, which is persistent storage for us to store our database.

Step 3: Add the connector block

Next, we’re going to expand upon this by adding connector. This block forms the HTTP listener we were looking for and receives the data from TTS.

We add this by adding the following to our compose file under services:

    connector:
        image: balenablocks/connector:raspberrypi3
        restart: always
        labels:
          io.balena.features.balena-api: '1'
        privileged: true
        environment:
          - 'ENABLE_EXTERNAL_HTTP_LISTENER=1'
        ports:
          - "8080"

Find out more about the connector block and the options it offers on the GitHub repo.

Step 4: Add InfluxDB

After connector, we need a database to store our data. For this we’re going to use InfluxDB. In the same way, add the following to the compose file, directly beneath the connector block. It’s important to ensure that the compose file is indented correctly.

    influxdb:
        image: influxdb@sha256:b4529fd53e9022bd1e83aaf04c6c6a1ebfffdfa112c31367d69da1354983a24b
        container_name: influxdb
        restart: always
        volumes:
            - 'database:/var/lib/influxdb'

Step 5: Add the dashboard block

Now we’ve got connector and a database provided by influxdb, we need something to visualize the data and allow us to see it with a web browser. Enter dashboard! This block provides Grafana in an automatically configured way, and we add this in a similar way two the other two services.

    dashboard:
        image: balenablocks/dashboard:raspberrypi3
        restart: always
        ports:
            - "80"
        volumes:
            - 'dashboard:/data'

Find out more about the dashboard block and the options it offers on the GitHub repo.

Step 6: Review and push

After combining all the above code blocks, you should be left with something like this:

version: '2.1'
volumes:
    database:
    dashboard:
services:
    connector:
        image: balenablocks/connector:raspberrypi3
        restart: always
        labels:
          io.balena.features.balena-api: '1'
        privileged: true
        environment:
          - 'ENABLE_EXTERNAL_HTTP_LISTENER=1'
        ports:
          - "8080"
    influxdb:
        image: influxdb@sha256:b4529fd53e9022bd1e83aaf04c6c6a1ebfffdfa112c31367d69da1354983a24b
        container_name: influxdb
        restart: always
        volumes:
            - 'database:/var/lib/influxdb'
    dashboard:
        image: balenablocks/dashboard:raspberrypi3
        restart: always
        ports:
            - "80"
        volumes:
            - 'dashboard:/data'

Open a command line or terminal and navigate to your project directory where you created docker-compose.yml. From here, and as long as you have set up the balena CLI tools, you should be able to run balena push <appName> where <appName> is the name of the application you created within the balenaCloud dashboard. This will combine the 3 blocks into an application which will then be downloaded by the Raspberry Pi.

Configure TTS

Once your device is up and running, we have to tell TTS where to send the data. First, enable the public URL on your device within the dashboard:

Open the public URL and you should see the Grafana dashboard provided by the dashboard block. Copy this URL from the address bar, append :8080 to the end after .com, and importantly change from HTTPS to HTTP, so that you have something that looks like this:

https://025c14d375b36dae49fd71771e83.balena-devices.com:8080

Head over to the TTS console, find your application and go to the ‘Integrations’ screen. From here you will be able to add a new integration and find the ‘Webhook’ option. Now add a Custom webhook.

Check for new HTTP integration back at TTS

Create the Custom webhook with the balenaCloud public URL on http pointing to the port 8080.

With this integration added to the TTS application, every time your node sends a reading, TTS will now forward it to your Raspberry Pi where the connector block is listening out for the data.

Access and customize the dashboard

Access the dashboard by accessing the device either via clicking the public URL link in the dashboard or by browsing to the local IP address of the device.

By default, the dashboard block adds all the data it receives from TTS to a dashboard called External Http Listener; it’s shown above under ‘Manage dashboards’ accessible via the dashboards menu on the left-hand menu.

Looking under fields called ‘payload fields’, you should see the data from your sensor!

Until next time

Thanks for following along with the process and reaching the end of this series. If you had success with the process I’d love to hear about it! Tweet us, tag us on Instagram or hit us up on Facebook. Similarly if you get stuck, or didn’t understand anything, let us know as well.

The best place to start is in the forums where I and other members of the balena team hang out and will be happy to chat balenaBlocks!