Dropserver Progress - November 2023

This is the monthly progress report for Dropserver for November 2023. The previous report is here.

The Big Picture

Although my main coding focus was getting Dropserver to install and update an application that is hosted on a third party website I got side-tracked by big thoughts on other topics. These don’t help the release tempo but are crucial for steering Dropserver in the right direction in the long term.

A Light-Bulb Moment

I had this crazy idea: Dropserver should install and run on MS Windows and MacOS as a GUI application. It wouldn’t be terribly useful in its default configuration (localhost only), but it would allow potential users to get a taste of Dropserver. From there, configuration wizards would allow them to make their install more useful: like connecting other devices and users via a WireGuard tunnel, or leveraging 3rd party or Dropserver services for obtaining TLS certs and DNS management (think ngrok, Tailscale, NextDNS).

The big idea is that you have to go where the users are, and they are likely on a Win/Mac machine. When they get around to using a dedicated machine for Dropserver, the spare machines they have are also likely to be Windows and Mac. And if they were to buy a machine for this, they will be far more comfortable using a Mac Mini or an Intel NUC running Windows than a Linux box. That’s just the reality of it.

By choosing Deno as the sandbox for Dropserver, I effectively do not have any platform constraints. I should take full advantage of that.

But it’s not just the Win/Mac thing, it’s also the idea that most people are GUI users. So instead of an intimidating CLI and config file based installation, users should be able to run an installer then use a GUI to set things up. With that in mind I would also try to support a Linux GUI distribution for those users who made the jump to the penguin OS but would still rather avoid setting up complex configs using the command line.

There are a lot of questions to answer before making this idea a reality, in particular lots of thought will have to go into making this as secure as possible. Thankfully it seems mainstream OS are getting the hint about sandboxing apps: Windows will sandbox apps for better security, and MacOS already has a Sandbox. These may allow me to preserve the second layer of sandboxing that I achieved using Bubblewrap on Linux.

A nagging feeling I’ve had with Dropserver these last months has been that it’s fine to have a platform that allows users to install web-apps easily. But if they can’t install the platform itself, what have I achieved? I think supporting mainstream OS with a GUI application is maybe a way to crack that nut. We’ll see. At least it gives me hope.

Frontend Thoughts

I am spending a bit of time thinking about what tech stack I would use if I were to rebuild the ds-host frontend. I am not thrilled with what I have right now (Vue 3 SPA) given that it is the control panel for a server. Why is it an SPA?

SPAs are great at rendering a widget very fast, without a round trip to the server if you already have the widget data locally (like maybe the widget appeared in a list view earlier in the session). The Leftovers and ShoppingList Dropserver apps benefit from being locally rendered because these are the kinds of apps where any waiting time is painful for the user. But is this necessary for ds-host?

I originally chose Vue 3 and made it an SPA because that’s how I built the previous thing, whatever that was. It was less effort to just use the tech I knew. But now I’m seeing that it may not have been the right call. The SPA offers a good developer experience for me when I work on the components, but the data management story is poor. See below for an example.

There is also lot of talk these days about web components and other more HTML-first approaches to app frontends. Maybe it’s a case of the grass is greener on the other side, but I can’t help but think about a rebuild that is more HTML-centric and simpler to maintain, not to mention more accessible and it could work without JS. The challenge is preserving the developer experience.

Progress on Installing Apps From URLs

Although I was able to fetch and install an app from a remote URL at the end of last month, that was hardly the end of the story. This month I implemented the storage and updating of the remote app’s info, then worked to update the frontend with this data.

Note that if you don’t see any activity in the Github repo, it’s because I’m working in the “app-from-url” branch.

DB work

The DB work was fairly straightforward but lengthy. There are lots of possible situations that arise from updating data fetched remotely: any of a number of errors from various steps (fetching, validating), permanent redirects, data that has changed, data that hasn’t changed. Sorting through all that took some time, and the implementation and tests felt like a lot of work. Still it got done: commit 61489f9.

Frontend work

Fetching the app URL data from the DB and displaying it in the frontend was no big deal. Classic CRUD work. A challenge arose when I realized that a user may never know there is a new version of an app because of the way the frontend data model works.

Simply put: the frontend loads all the apps and appspace data when a list of these is viewed, and keeps that data around between page navigation. This means that rendering an app detail page is instant. Neat. But miserable. Holding on to data means you have to update it when it changes by pushing from the backend.

I started doing work to push events to the frontend: I implemented an event handler for app URL data on the backend, which ended up being my first contact with Go Generics. It took a bit of time but I found a satisfactory way of making backend events easy to implement thanks to Generics. I’ll call that a win. Commits 6e6f0d and e0e249.

In the end I am not pushing events to the frontend. I think the whole ds-host frontend is built on wrong-headed ideas given the type of data it presents. It just doesn’t need to be an SPA. So instead of adding complexity I just made it such that app URL data is reloaded when a user clicks through to a Manage App page. This solves the problem well enough and easily enough.

Refresh app URL data

In commit 0a7c31 I tied a bunch of things together (the DB data, in particular the stashed ETag, the frontend, and the fetch code with SSRF protection) to enable one-click refresh of the remote app URL data. Whew! It’s annoying that this took so long but it’s progress.

From there I shouldn’t have much trouble automatically refreshing the remote app URL data on a timer, which ds-host will do about once per day.

Next up

Next I’ll have to make my dummy remote app sites that I use to test all this stuff so that I can change the site dynamically and then fetch an update. The reason is even though I have all the pieces in place I need to implement a proper UI for when a new app version is available or other edge cases. Once I have a good UI for handling a new version of an app, I’ll make it so ds-dev can generate a static site for app distribution, and then I’ll ship 0.13.

I’ll also continue to research and develop my ideas about frontend technology and an easily-installable mainstream-OS GUI version of Dropserver.