This is the progress report for Dropserver for May 2024. Here is last month’s report.

May was a “Spring cleaning” month. Before launching into my next big project (Tailscale integration) I want to have a clean code base, or at least clean out the obvious relics of ancient ideas that never fully blossomed.

Here we go:

Remove Unused Appspace DB Code

Back in ancient times Dropserver apps could store data in a sqlite DB that was managed by Dropserver. Reads and writes went through a rather lame API from the sandbox side via a decidedly unfinished protocol. I disabled this a long time ago and recommend app devs use a WASM build of SQLite directly in their app.

Yet there was quite a bit of code left over from this “Appspace DB” that remained to this day. No more!

Cleaning this up was important because the Appspace DB code is easily confused with the “Appspace Meta DB” which holds data about the appspace and is going to get a lot of attention when I work on Tailscale. integration (see below).

Rethink Backwards Compabilitlity

Tailscale integration will bring new data types to the Appspace Meta DB, where users and their access types and ids are stored. This means I’ll have two versions of appspace meta DBs in the wild: the original version, and the new one with the ability to store tailscale data.

This launched me down a path of rethinking how to do backwards compatibility in Dropserver. Its important to get it right, and I’m pretty sure I got it wrong, so a rethink is in order.

See the post Building Backwards Compatibility into Dropserver for the latest thinking.

I don’t mention the Appspace MEta DB in the post, but the implication is that there is only one version of the Meta DB understood by a given version of Dropserver.

As such, the host will migrate the Meta DB of any appspace it encounters to its current version.

This means that if you upgrade ds-host and it supports new features and a new MetaDB schema, then all the appspaces get migrated to this schema. If you import a backup of an appspace and it needs to be migrated, that will happen too.

Should you want to use an appspace that was migrated on an older ds-host, then down-migration tools should be available too, possibly within ds-host itself.

With that in mind I was able to eliminate a lot of v0 code, which makes the codebase feel cleaner. I still need to write the Appspace Meta DB migration logic though.

Deno Config and Import maps

In addition to cleaning up the code, I have to keep an eye on the evolution of Deno to make sure I don’t fall too far behind.

With the advent of JSR I am feeling more pressure to support custom import maps for Dropserver apps. JSR leans on import maps to make adding packages more convenient, and I should support that. I spent some time exploring all the ways Deno uses import maps, and looked at the spec carefully and ran a number of experiments.

Dropserver uses import maps to prevent static access to files outside the allowed directories (see this issue comment). Allowing custom import maps means being careful not to give the app any ability to read a file outside of its direcotry.

I’m happy with what I learned and I almost started implementing this but something happened that made me put it aside for now.

File Descriptor and Goroutine Leaks

My personal Dropserver instance stopped working all of a sudden. Turns out I had too many open files? This was frustrating because that is not the first time it happens and I thought I fixed it last time. Evidently not.

Dropserver is supposed to run for a long time without needing any attention. When it blows up like that it’s really disruptive. These kinds of headaches are why people don’t want to run their own services. It’s a pain to have to deal with them when they don’t run.

My hunt for leaking file descriptors and goroutines was more thorough this time: see commits 6e0f48, 1dd6bc and 9f442f. I might write a blog post on what I learned along the way.

Release Dropserver 0.13.2

By now I’ve accumulated a number of minor improvements and code changes, so it would be good to push that out to the world.

Prior to shipping, a routine upgrade of node modules turned into a hour-long battle with my Tailwind and PostCSS and package.json files. I swore off Tailwind a while ago, but I have yet to rewrite the Dropserver frontend using regular CSS, so I’m subject to it ruining my joie de vivre for now. (The error was “Vite CJS Node API deprecated”; see this commit for the fix.)

Version 0.13.2 is finally out, it includes a proxy for outbound requests from the sandbox that I described last month, the fix to goroutine and fd leaks, and a few other things.

Compile ds-host for MacOS

For the Tailscale integration work I thought it might be easier if ds-host was running directly on my Mac. ds-dev already runs fine on the OS, so why wouldn’t ds-host?

One little commit later and ds-host runs on the Mac too! With the config set to just serve over localhost, I can access it via dropid.localhost and appspaces are accessed at their own subdomain like leftovers.localhost (Except on Safari, which doesn’t support subdomains on localhost 😠).

This is encouraging. I’ll need to make some tweaks, and try to get ds-host to run under the MacOS sandbox since Bubblewrap doesn’t work in this environment (On Liunx Bubblewrap is a second layer of sandboxing in addition to Deno) but I can foresee a future where I offer a build of ds-host for the Mac.

I am so excited about this I checked the price of used Mac Minis: under $100 if you’re willing to live with 10 year old spinning disks! Might want to spend a bit more, but self-hosting Dropserver on a Mac Mini would be a great option for people who know Macs and are intimidated by Linux.

Onwards

I just started work on the Tailscale integration. A tailnet is a rather different environment than the global internet, and as such a number of assumptions I naively baked into the Dropserver code are going to have to be rethought. So this is going to take some time.

In the meantime, I just looked at my Grafana dashboard on my personal instance and the goroutines and file descriptors lines are staying under control. Pfew!

(Although I know of some situations where I still have problems, like upgrading an app causes leaks. That’s for another day!)

See you next time.