Open source resin.io: progress and next steps

In 2015, we announced our plans to create an open source form of the core resin.io product, with the goal of providing an easy to deploy platform for managing device fleets. Last summer we reiterated on our commitment, and in our status update we stated that, given our progress and the remaining work, the end of 2017 sounded like a reasonable date.

Our timeline expectations haven't quite matched our reality, but we haven't been sitting idle. In the last few months, we've released a stream of frequently requested product features, built our own IoT-optimized container engine, and revamped the entire platform to support multicontainer applications. While we'd love to say this post is announcing open source resin.io, we're not quite there yet, but we do have some updates on where things stand and what steps lay ahead.

The resin.io platform broadly comprises three parts: on-device software, such as the operating system, device agent and base Docker images, client tools and frameworks that let users interface with the platform (the CLI and SDKs), and the cloud infrastructure that brings everything together. With the device and client sides already fully open sourced, our focus has been on the various backend components and their interactions.

Let's take a look at what we've accomplished on that front since our last update:

First and foremost, we have open sourced PineJS, our custom API framework, and its related core dependencies. This is the first step towards our goal of open sourcing our entire backend API. It took us some time to open source PineJS, but the process had several extremely valuable side effects. To make this framework accessible outside of our organization, we had to improve its usability and documentation, as well as evaluate the boundaries that define how an application interacts with the framework. We'll continue to improve PineJS, including the documentation, but we're happy you can get started with the basics.

Another area where we've made great progress is in simplifying our deploy pipeline. Using git to deploy code to devices—the ubiquitous git push resin master— provides a low bar of entry to resin.io. Devices, however, do not run code directly, but rather a product of that code (in the form of a Docker image). We've redefined our deploy pipeline to be agnostic to how an image is built and updated our CLI and builder to use these new interfaces. git push resin master is now just one form of input to the platform. What this means for open source resin.io is that deployments can now be performed without the presence of the git server and builder, allowing us to bundle two fewer components, and bringing us closer to our goal of an easy to deploy open source platform.

As we improved our deploy pipeline, we also performed a major refactor of our cloud builder and open sourced most parts of it. We've included these pieces in our CLI, so the CLI's build and deploy capabilities are now aligned with the cloud builder and steadily receive improvements. We removed the CLI's dependency on the cloud builder to upload an image to our registry—image uploads are no longer a privileged operation that only the builder is allowed to perform, and the platform can now authorise individual clients acting on behalf of the user. With additional features made possible by the fact that the CLI runs on the user's computer, such as local build caching and project source code privacy, the CLI now makes a great alternative to our cloud builder and a first class frontend to the platform.

Though we still have some way to go before the platform is open sourced, the remaining work is now technically well-defined. We have audited every component and workflow across the stack, discussed and documented in great detail the areas that still need work and the hooks we must put in place, and specified a clear set of tasks and milestones we must accomplish to get there.

Here's a brief look at what's next and what we're currently focusing on:

Each device supported by resin.io is described by a device type, which is a JSON structure with values for the device's architecture, name, and configuration options, among other things. This information is used by the API and clients to perform functions such as configuring an image for flashing on a device. Currently, device types live alongside the respective device images in a mostly ad hoc fashion. This means that the open source platform would have to maintain intimate knowledge of resin.io-specific functionality. We are standardizing how such device metadata and images are stored and later discovered by the platform.

Last, but no less important, is our continuing effort to open source the API. Our API evolved from a monolith into a web service with well-defined responsibilities. It is the backbone of the platform, acting as the single source of truth and mediating many workflows and operations. Not all parts of the API can or should be open sourced, however. Some parts are irrelevant to the open source platform, while others support features that define managed resin.io and will stay proprietary. Therefore we're incrementally building bridges to allow resin.io to extend the open source core where appropriate.

One clear goal we've set for the open source platform from the very beginning is that it must not be a fork or a project that is otherwise run in parallel to resin.io. Instead, it must form a coherent product with enough extensibility baked in to allow a cloud service like resin.io, with all of its added value, to be built on top. In other words, resin.io will be the first production deployment of the open source platform, and it must transition to that state incrementally, without service interruption as changes are absorbed.

We strongly believe that this approach will be beneficial to the open source platform, as well as to resin.io itself. It will result in a much more extensible and robust platform than any ad hoc implementation, by virtue of needing to support the large scale deployment of resin.io from day one. We recognize that this is a great opportunity for us to improve our infrastructure, provide new or improved abstractions, and clean up technical debt, resulting in a leaner platform that allows us to iterate faster. Therefore, we're as eager as you, or even more so, to deliver an open source platform that is reliable, flexible and easy to use.

So when do we expect to deliver? We conservatively expect to announce open source resin.io by September of this year. After going through a phase of demanding architectural changes for the multicontainer feature, we believe the stage is now set. Open sourcing is our next top priority and, perhaps most importantly, there's now a team dedicated to driving the project forward and ensuring a timely release. We're looking forward to getting an open source version of resin.io into the hands of our community, and we can’t wait to see the contributions you’ll make and the projects you’ll build with it!