We recently released a feature that gives you a lot more visibility into your remote devices running on resin.io. For those with resinOS versions 2.7.5 and above, it is now possible to SSH into the host OS. This can be done directly from the dashboard or through the CLI. Host OS SSH access gives you a handful of tools that can help you gather more information about potential issues. To help you get started, here are a few troubleshooting tips:
1. Check logs
journalctl
Information from a variety of services can be found using the journalctl utility. As the number of journalctl messages can be quite large, it is good to know how to narrow your search.
To find messages from a specific service, use the -u flag:
journalctl -u systemd-timesyncd
To return the last x messages, use -fn x:
journalctl -fn 100 -u resin-supervisor
dmesg
For displaying messages from the kernel, you can use dmesg. Similar to journalctl, dmesg may have an unmanageable output without some additional commands:
dmesg | tail -n 100
2. Monitor balena
ResinOS, beginning with version 2.9.0, includes the lightweight container engine balena to manage Docker containers. If you think the supervisor or application container may be having problems, you’ll want to do use balena for debugging.
This command will show the status of all containers:
balena ps -a
You can also check the journalctl logs for messages related to balena:
journalctl -fn 100 -u balena
For devices with resinOS versions earlier than 2.9.0, you can replace balena in these commands with docker.
3. Inspect network settings
NetworkManager
NetworkManager includes a CLI that can be useful for debugging your ethernet and WiFi connections. The nmcli command, on its own, will show all interfaces and the connections they have. nmcli c provides a connection summary, showing all known connection files with the connected ones highlighted. nmcli d displays all network interfaces (devices).
Another useful place to look for NetworkManager information is in the journalctl logs:
journalctl -fn 100 -u NetworkManager
ModemManager
Similar to NetworkManager, ModemManager includes a CLI, mmcli, to manage cellular connections. mmcli -L provides a list of available modems.
4. Look up version information
Knowing what version of a specific service is being run on your device can help you troubleshoot compatibility issues, known bugs, and supported features.
Many services provide a direct option for displaying their version:
udevadm --version
systemd --version
openssl version
5. Understand the file system
In some cases, you may need to examine the contents of certain directories or files directly. One location that is useful for troubleshooting purposes is the /data directory, which contains your device’s Docker images, persistent application data, and host OS update logs. The /boot directory includes configuration files, such as config.txt and NetworkManager connections.
Note that the filesystem layout may look slightly different from what you’d expect—for example the two locations mentioned above are found at /mnt/data and /mnt/boot, respectively.
6. Use the forums
If your troubleshooting runs into a wall, you can always reach out to us on our forums. When you do, make sure to include as much relevant information as possible. This includes output from any of the utilities mentioned above (after inspecting for sensitive application information or environment variables), as well as resinOS version, supervisor version, and device type, which can all be found on the device summary page of the dashboard.
Not using resin.io yet? Sign up today to deploy and manage applications across fleets of connected devices. Your first 10 devices are always free!
