Building Backwards Compatibility into Dropserver

May 20 2024

An unfortunate pitfall of modern computing is to be forced to make the choice between upgrading an operating system and continuing to use a beloved old app. Forcing this on users brings out some choice words for the developers of the OS, but a system that can evolve to its full potential while running old code is hard to build.

I would like for Dropserver, an OS of sorts, to continue to run old apps for as long as possible even as it evolves. Here is how I’m thinking through it.

Layers

There are several layers of APIs between the application code and the core Dropserver internals. These layers provide opportunities to make backwards compatible adapters, or to hide the ugliness of aging legacy APIs. Here is the list of layers as things stand:

dropserver_app library. Imported by the app to interact with the Dropserver system, as shown in the tutorial.
dropserver_lib_support compatibility layer is a dependency of both dropserver_app and denosandboxcode. It’s a bunch of TypeScript types that ensures the layer above and the layer below can talk to each other when they are run together.
denosandboxcode ships with the Dropserver host, and is therefore part of its repo. It runs in the sandbox alongside the app code. It communicates with “Sandbox Services”.
“Sandbox Services” in Dropserver host. It listens for queries from the sandbox side and passes these on to the appropriate data model or whatever.
Data models and controllers in Dropserver host. These talk to databases or other forms of storage. It’s the end of the line.

Let’s look at how each layer help backwards-compatibility:

dropserver_app

The dropserver_app library can evolve without changing the underlying API. In other words, I can make improvements to the syntactic sugar without making any changes to the underlying APIs. As such there are no changes to be made that may cause older versions to no longer work.

However making changes to this library can put app developers in an annoying spot as we’ll see below.

dropserver_lib_support

dropserver_lib_support is the compatibility layer that must remain back-compatible with itself. If I want to change anything there, I have to add a new type, or a new optional field in an existing type. I can’t delete anything, ever, if I want old apps to run. It’s going to grow over time and will include bad function names like “getUsersV3”. However, because lib_support is not used directly by the app dev, they are not subjected to this. The new versions of dropserver_app will use the new APIs with nice names.

Again since dropserver_lib_support is hidden, it can be low level and it can be “ugly”. That’s fine, it’s hidden away. Types should be added slowly and carefully with an eye to maintaining them long term. The syntactic sugar and nice API can be left for dropserver_app.

denosandboxcode

In denosandboxcode I have the opportunity to handle old APIs by transforming them into new APIs. Remember that denosandboxcode ships with Dropserver. As a result it handles the latest API (at the time of shipping). But it also has to handle all the old APIs. The best possible outcome is that old API calls can be transformed into new API calls right there in the sandbox. That way the Dropserver host does not need to ship with a “Sandbox Service” that can handle multiple versions of an API.

Sandbox Services

It’s possible an old API can’t be coerced into a new API inside the sandbox for some reason, in which case the transformation may be able to take place at the service level. The difference between this and the sandbox is that the “Sandbox Service” runs on the host and has access to all the data about the appspace, users, etc… while that information is strictly limited inside the sandbox.

Core Models and Controllers

Finally, if all else fails, there is the possibility of versioning the very core of Dropserver’s handling of app and appspace data, like the users DB, etc… This manifests itself with parts of the host code named v0AppspaceRoutes and v0AppspaceUsers (note the v0.)

This was actually going to be the primary way of doing things but my thinking has evolved.

The Thinking Evolves

I originally set up a good chunk of Dropserver code with these v0 names. I did this thinking it was the right approach, but I just removed these. Here is why:

Versioning is viral: when you start having versioned data structures, some other structs that depend on them may need to be versioned as well. If you display any of this data to the user, the frontend code that handles this gets versioned too, not to mention the APIs used to fetch the data. Next thing you know you’ve got more versioned code than you ever wanted to see.
More code to maintain is bad: maintaining old code paths is hard. Keep these to a minimum. You do this by eliminating the difference between an old API and the current one as early as possible. That means inside denosandboxcode if at all possible. That way the core of Dropserver stays free of versioned madness.
The goal should be no changes anyways. API versions imply breaking changes. Nobody wants that. So why lay the groundwork for it?
Makes small additions hard. If I want to add a small feature, do I have to increment the version? That makes a big deal of every small change. I might even hesitate to make the change, holding it back until I have all the pieces that warrant a major API change. This doesn’t line up with how I’d like to work.
API versions imply big releases. A project like Android has API versions, and new versions coincide with new Android releases, which are announced at big flashy events every year. That’s not how I envision Dropserver. It’s a stable project. It must evolve of course, but not in that big flashy release manner that seems to be favored by companies that build planned-obsolescence into their products. By contrast, look at how LXD do it: they have a 1.0 API, and a myriad of API extensions. I like that better.

For all the reasons above I’ve eliminated the v0 code wherever I could.

What About Forward Compatibility?

One conundrum I still have to deal with is the situation where a new version of Dropserver includes a new API for app developers, and therefore is accompanied by a new version of dropserver_app. If an app developer uses the newest dropserver_app library to write a new app, does that app require the latest version of DS to run? Even if it doesn’t use the new API?

I’ve talked about backwards compatibility where an old app can continue to run on newer Dropserver versions. But what about a brand new app running on an older instance?

While I would like Dropserver instances to be updated regularly, realistically this won’t happen. Ideally a new app made with recent versions of the libraries I publish should be usable on older versions of Dropserver.

The solution here I think is to not have a monolithic dropserver_app.

C’Mon Man Just One More Layer

I haven’t fully worked this out but here’s where I think things are going to go:

dropserver_app will become a shell of sorts, and the app dev will import other libraries, like dropserver_users and dropserver_routes to interact with the DS system. The objects created by these libraries are passed into createApp().

So if an app only needs to use basic (older) route features, they can select the dropserver_routes@^1.0.0 and keep their app usable on old versions of Dropserver. If they need the newest features, they can use ^2.0.0.

createApp() can examine the objects passed in and determine the APIs that are required to run this app. This list of APIs used by the app is very handy for devX and UX reasons:

ds-dev can show the app developer the minimum version of Dropserver needed to run their app (you can assume Dropserver will have a map of API -> minimum Dropserver version)
When an app is published, that info can be embedded in the manifest and shown in the app distribution site.
When installing, the user can reliably be told the version of Dropserver they need.

Another advantage with this approach is that the app developer can use the latest version of the dropserver_routes without being forced to use the latest version of the users API (since they are separate libraries.) This means they can adopt a new feature without being cornered into adapting unrelated parts of their code.

Let’s Get Real

Creating a solid foundation for backwards compatibility for a platform that is not even off the ground yet is one of the more challenging aspects of Dropserver. I think it’s important though as it’s the kind of think that is hard to fix down the line.

This challenge is not academic. I have a bunch of Dropserver apps in various states of development, and I currently depend on four different apps on a daily basis. I’d hate to release a new version of Dropserver that would force me to update all of these before I can upgrade my personal instance.

I’m sure I’ll have to make more adjustments to my approach, but I think at a fundamental level the presence of multiple layers gives a lot of opportunities to fix problems surgically instead of needing to do big rewrites.