Dropserver Progress - December 2023

This is the monthly progress report for Dropserver for December 2023. The previous report is here.

The Big Picture

I made a big push to get version 0.13 out the door. While I didn’t succeed at actually cutting the release, the most challenging part is complete and I am now just adding a few more items. I also continued developing thoughts on how to take Dropserver to the next stage. I started work on getting ds-dev to work on the Windows operating system.

Work on Installing App from URL

Version 0.13 will make it possible to add an app to your Dropserver instance by pasting a URL. The app will be downloaded and installed, and Dropserver will periodically query that URL to determine if there are new versions of the app available. The user will see a callout when a new version is available to be installed.

ds-dev will generate all the necessary assets and JSON files necessary to distribute your app via a URL. All the developer needs to do is upload the generated files to a static site server.

Getting all these pieces to work took some doing. In this progress report I’ll spare you the play-by-play of all the refactorings I had to do. Below are a few notable thoughts about recent work:

The App Distribution Test Site Became a Dropserver App

Back in November I was using an entirely static site to test app distribution. I had created (by hand because the site generation code was not built) a static site that offered the necessary JSON and other files to distribute a Dropserver app. I uploaded them to Netlify and was able to test ds-host’s ability to fetch a remote app by pasting that URL in the UI.

However at the end of last month I got to the point where I needed to easily make changes to a published app in order to test how Dropserver displays the multitude of states it finds on the remote. For example: if I am writing the code that handles the appearance of a new app version on the distribution site, then ideally I should be able to set the distribution site to show that new version or not. That way I can repeatedly install the app, then “publish” the new version and see how ds-host behaves.

It is quite easy to update the site published on Netlify, but it’s still too slow and tedious for something I need to do dozens of times per day.

The solution was to create a Dropserver app that could serve whatever situation I could dream up. I didn’t want to spend much time on this bit of tooling, so it only lets me select from a menu of pre-coded situations.

The app is utterly basic (the main interface is an HTML form, with no styling or JS) and leverages some of Dropserver’s capabilities, like private routes for selecting the situation, and public file server route handlers for serving the JSON data files and app packages.

This worked really well. It was also a good exercise for me and for Dropserver: each time I thought of a new situation I wanted to serve, I coded it in the app, incremented the version, packaged and uploaded it to my instance, and it was live in a few seconds. This was accelerated dogfooding for app packaging, installation, and appspace migration.

With the ability to switch the responses served by the app distribution site, I was ready to grind through the rest of the project.

Detour into The Future

While doing the work described above it was hard to resist the temptation to create a full blown Dropserver app that serves as the app distribution site, instead of the rudimentary hard-coded thing. In the future, this is clearly how it will work: I will publish an app that you can use to publish your app.

The motivator for this is that static sites can be a bit limiting. For example perhaps I’d like to get changelogs for versions 1.3.0 to 2.1.1. With a static site I’d have to do a lot of requests to get versions one-by-one, or perhaps I can load the entire app changelog which could be a lot of data. With a server handling the request, this becomes a single request that returns just what is asked.

There are other reasons for the app distribution system to be Dropserver-based: it could include a support forum, reviews and comments, etc…

However the reason for sticking with the static site idea for now is that I think many devs will initially experiment with Dropserver locally, and may not expose their instance to the broader internet. As a result there should be a way to publish Dropserver apps without using Dropserver.

The Grind

A lot of the work went into correctly reflecting the data obtained from the remote app distribution site in the the local DB and in the UI. Some examples:

• Dividing the UI flow between apps installed by uploading a package and apps installed from URL. When you add an app by uploading a package, newer versions must be obtained the same way. And vice versa, and app from a URL can only be updated when a new version is published at that URL.
• Lots of refactors to support re-using the same code paths for different but similar use cases, like fetching a changelog for a new app and a new version, validating manifest prior to fetching package as well as during installation, etc..
• Lots of head-scratching regarding error handling and reporting: the same error can be fatal for one process but informational in a different situation. This makes dividing up the code a bit tricky. I also improved how I report warnings to the user.
• Reflecting as much of the app to the user before they even fetch the remote package. So lots of new routes, remote fetches, and UI components to display all this as needed.

Windows Version of ds-dev

Last month I decided to make Dropserver runnable on more common operating systems in an attempt to reduce the installation hurdles for self-hosters. As a first step I’m trying to get ds-dev to work on Windows.

So far I managed to get the test suite to run by eliminating the Linux/unix-specific code. At some point though I’ll have to figure out how get ds-dev and ds-host to quit nicely since Windows doesn’t support SIGTERM.

Now I’m just going through the many failing tests and fixing them for Windows. I don’t know if I’ll hit any show-stoppers. I found one reason to be optimistic: Windows supports AF_UNIX sockets! Dropserver communicates with its Deno sandboxes via unix sockets. Not having to re-write that is a plus!

Up Next

In January I’m shipping 0.13. That’s my focus. After that, we’ll see. See you on the other side!