Dropserver Progress - April to June 2023

I am going to try writing monthly updates on the progress of Dropserver development. Regardless of whether I have anything significant to share, I’ll post about the past month’s work. I’ll look at my commits and my notes (I take copious notes, arranged chronologically and in threads, but that’s the subject of a different post) and summarize what I worked on.

Since this is the first such post it will cover all the work I’ve done from my last release in mid April until the end of June 2023.

The Big Picture

I am working on application packaging. The goal is to make it easy for someone who wrote an app for Dropserver to make it available to others, and for non-technical people to make use of that app in their Dropserver instance.

The current version of Dropserver allows a user to upload a directory of app files. This is error prone and makes it hard to share apps around. Furthermore, the so-called “manifest” has very little useful information in it (basically just app name and version).

Here is what I had to do:

• come up with a package format, including the manifest and the archive and compression format
• create the tooling that creates the package and integrate it into ds-dev
• adapt ds-host to accept these packages and install the app
• update the frontend of ds-host to show app package values and warnings

This has been a big project. I haven’t released a new version of Dropserver since April 10th 2023, but hopefully I should be able to release this sometime in July.

Details

Note: I am developing this in the app-package-1 branch.

Exploratory thoughts

As is typical with how I work, I first had to explore the outer fringes of reasonable ideas before accepting that would I have to build is pretty straightforward and follows in line with prior art.

Do I even need a manifest?

What’s so necessary about a “manifest”? Dropserver already runs the code to get metadata about the app (such as available migrations) and perhaps putting everything in app code is the simplest thing to do for developers?

Unfortunately this doesn’t work for some fields: the entrypoint of the code can’t be set in code! And even if all data could be obtained by running code, Dropserver would still need a manifest format to stash that data. You do not want to run a sandbox every time you want to know something about an app. Once you have that, why not let the app dev write into it directly instead of cluttering their code?

“Single File Apps”

Wouldn’t it be great if you could write a single TS/JS file and have it become a web-app without doing any packaging work. You could share it around as a GitHub Gist or anywhere else. (Mastodon post)

That sounds cool but the reality is that most web-based things are inherently multi-file. If you serve a frontend, even a simple one, you’ll want HTML templates and a style sheet. I realized the value of “Single-File Apps” was low and the additional complexity in Dropserver was not worth it. Instead I am focusing on making packaging easy thanks to built-in tooling in ds-dev.

App distribution site

I don’t want Dropserver app devs and users to be beholden to a single centralized app store. It should be possible for an application developer to publish their app on a website that they control, and users should be able to install it from there. The independent and open web demands this.

I had originally thought that such an application distribution website would actually be driven by a Dropserver application, but at this stage it’s better to do something simpler. An app distribution site will just be a static site hostable on any static website host.

This is one idea I have not abandoned. I’m merely punting on it for a future version, but it’s still the plan.

Figure out the manifest

Having decided I needed a manifest, I had to decide what fields would go in it, what format these fields would be, and how they would be validated. I created a GH Project and an Ethercalc table to work through the scenarios and possible fields. (Note that these are outdated at this point and do not reflect all my latest thinking.)

I looked at manifests for Android (not very relevant), NPM’s package.json, Sandstorm, YUNoHost and others for inspiration. All were informative but I found some cool ideas in YUNoHost, like pre-defined tags for anti-features and a system for determining application quality levels. I’m not implementing these ideas yet, but I may in the future.

I had to decide on TOML versus JSON for the manifest file (mastodon link). I decided to use JSON. TOML is lesser known and still has detractors, while JSON is nothing new for JS devs, even if it’s not ideal. Additionally, the packaging process writes its own version of the manifest, so JSON is a bit easier to work with.

Package format

Zip or tar.gz or something else? See this Mastodon thread. I found tar.gz to be a better bet. It’s well supported and well understood. It’s fairly safe. No patent questions, unlike Zip.

Data backups of appspaces are saved in Zip format right now. But that’s because I expect end-users to want to open these locally. There is no reason for a non-dev to manually open the archive of an app, so it’s OK if it’s in a less popular archive format.

Packaging flow

I worked out that the app developer would write a partial manifest. Currently the only absolutely required field is the app version. They create their app package using ds-dev -create-package which runs the sandbox to obtain additional data like data schema, available migration functions, and others. These get added to the manifest, and this “augmented manifest” is placed in the package instead of the dev’s original. Additionally the augmented manifest is saved locally.

I do this because I want the dev to write a minimal manifest if possible, but some fields should be available without having to run a sandbox. So the augmented manifest serves as a cache for values determined at runtime.

Installation flow

A user can install an app by uploading an app package to their Dropserver account (wherever it may be). I wrote the necessary pieces to enable this flow:

• ds-host unpacks the archive carefully to prevent archive-based attacks. In particular it has an expectation of a maximum size for a package, and bails on the process as soon as it hits that maximum. The max is hard-coded right now, but it will tie in to the user disk quota mechanism when that gets written.
• It reads the manifest and runs the sandbox to obtain the values that are available at runtime. It validates all the values (note that it uses the values it gets from the sandbox instead of the values of the manifest where it can.)
• If the user commits the new app, it saves the extracted files, and it places the augmented manifest in a JSON column in the DB. It also keep the original package file.

Add fields to the manifest

After I got rudimentary packaging and installation done, I added handling for more fields of the manifest:

• App entrypoint: This is known as main in package.json. It’s the file that the sandbox runs. It used to be hard-coded as app.ts. Now it defaults to app.ts and app.js, or to whatever the manifest says. Here I had to be careful that it is not possible to make a path that climbs out of the app package.

• App icon: This will be shown alongside the app name in ds-host UI and helps differentiate the apps. The app icon value in the manifest is a path relative to the package root to an image file. Naturally I had to take care that this would not be used to read files that are outside the package. I still have to put constraints on image size in pixels, as well as file size. Eventually expect to use something similar to Android’s Icon spec to guide icon design.

• Accent color: A color that the dev chooses that complements the app icon or matches the UI of their app. This color is used to decorate a box that gives information about the app in the ds-host UI. It’s never used as a background or foreground color for text so it does not affect accessibility. To make things easy for the dev any CSS color is accepted. I found a CSS color parser in Go which made that nice and easy.

• Authors: An array of author objects each with name, URL and email.

• URLs: I added fields for app website, code repository and funding. Then I realized that there is such a thing as the javascript: protocol, and just displaying these URLs as links in the ds-host UI could be a terrible idea. So they get validated as proper http(s):// URLs. If they fail they get removed from the manifest upon app installation. I process author URLs the same.

• Release date: A date in GMT that is automatically added to the manifest at package creation time. Its sole purpose is to let the user know whether the version was released recently or not.

• License: I decided that the license field should preferably be an SPDX string. If it’s not you get a warning. The reason for using SPDX is that I may want to give users a quick synopsis of the license but I can only really do that if I can correctly identify it, and SPDX is really made for that. Furthermore in the unlikely event that a license is incompatible with Dropserver, then being able to positively identify it will be useful.

ds-dev frontend

I updated the UI of ds-dev, the local dev-oriented app runner, to show all the values and warnings about the app and the manifest that a user would get if they installed the app on ds-host. This should make it easy for the app dev to correct any issues before shipping.

ds-dev runs the same code to validate the app as ds-host does when it installs the app (as does the -create-package command) therefore the developer sees all the warnings the user will see (with the caveat that different versions of these programs may behave differently).

App version model

With so much metadata stashed in the manifest, I had to figure which fields were needed and in what circumstances. I don’t want to return the entire manifest every time some function needs to know something about the app version.

After spinning my wheels for a while I figured that there was an internal use case that is used to run the sandbox and do other operations, and only needs values related to that (entrypoint, and a few others). Then there is the basic “UI” case, where we are presenting the app to the user and need short-description, authors, urls, icon, color, etc… Finally we sometimes do need to get the entire manifest so there is a method for that.

In the first two cases I make use of sqlite’s JSON functions to cherry-pick desired values from the JSON manifest (which is stored in a JSON column in the db). I am happy with this. It means I have the whole manifest available, or any subset of it that I may need.

ds-host frontend

I did a lot of work on the ds-host frontend. With data like icon, color, license, authors now available as metadata for the app I was able to make the display of apps more rich and interesting.

See screenshots in this Mastodon thread.

Punting

I had to punt on a lot of things. It’s hard to do but it’s necessary to keep momentum. I’ll tackle these later:

• signatures for app packages: needs to be done right, I want to spend more time thinking about it.
• internationalization of manifest values: I need a Dropserver-wide strategy for I18N, which I’m thinking about but it’s not there.
• release notes: surprisingly tricky to do well. I could take the easy way out (one md file with all release notes), but I want to see if I can improve on the status quo here.
• get app from URL and auto-updating: that’s another huge chunk of work that I will do in a different release.

Looking Ahead

I’m at the point where I’m finishing up loose ends and doing testing. I still have a lot of docs to write but I hope to release this soon.

After that I’d like to go straight to work on installing an app from a URL, and everything that comes with that, but I may have to take a step back to see if that’s really what needs to happen at that point in time.

In the future I may add a number of other fields to the manifest, such as:

• age appropriateness
• anti-features (like YUNoHost)
• features / capabilities listing (like ocdtrekkie suggested)
• package size / installed size
• whether additional resources are needed or if the package is self-contained

Other big ticket items that are in need of attention for Dropserver as whole:

• I18N
• Appspace user management and instance-to-instance data sharing for users
• More powerful app capabilities: enable outbound requests (with permissions), support Server-Sent Events, and many more…
• Working towards a hosted version of Dropserver
• And countless other stuff…

Lots to do. See you next month!