Dropserver Progress - October 2023
This is the monthly progress report for Dropserver for October 2023. The previous report is here.
The Big Picture
My focus is on getting Dropserver to install and update an application that is hosted on a third party website. Most of the work is really just thinking about what the endpoints look like on the third party site, and all the different ways a user will proceed through the steps of installing an app and create an appspace, in particular thinking about how unattended upgrades will work.
I also worked on other things as they came up, which included releasing version 0.12 early.
Vagrant file for ds-host
Jacob Weisz submitted a PR for a Vagrant file to help potential developers build and run ds-host on their machine. I merged the PR of course but it made me wonder how Vagrant could be leveraged to really make it easy for developers to contribute to Dropserver.
Someone contributing to ds-host should be able to quickly rebuild and relaunch ds-host after making changes. Running ds-dev, compiled from source, should be easy as well.
A frontend developer should be able to launch this Vagrant machine and quickly get to a point where they are editing Vue components while enjoying the benefits of Hot Module Reloading.
I’ll have to dive into Vagrant a bit to make all this happen. I will have to dogfood this too: I will ditch the hand built VM I currently use to run ds-host and use the Vagrant development environment to do all the development of Dropserver.
Anyways, thanks Jacob!
Bug fix and 0.12 comes out early
Trevor Flowers reported a panic in ds-dev while working on the tutorial app. Never great to have a panic on what should be well trodden code paths of a “Hello World” app. It turns out Vim puts a file called “4913” and immediately deletes it. This caused a panic in my application code watcher loop.
It’s a classic case of a developer putting a panic in a branch of code because they think “this can’t ever happen” and a user comes along and says “oh yeah? watch this!".
This was quickly fixed (explanation and resolution in this nice blog post) but unfortunately I had done quite a bit of work in the main branch of Dropserver, including a proper new feature (app changelogs) so that a patch increment (0.11.1) was inappropriate.
Thus was born Dropserver version 0.12, which frankly is no bad thing.
Going forwards I will be more careful not to mistreat the master branch so that I am always ready to release a patch if the need arises.
Thank you Trevor for taking the time to try working with ds-dev and reporting the bug.
Work on Apps from URLs
The early part of the month was spent figuring out how this should work. It’s not that getting an app from a URL is much harder than installing an app that was just uploaded. It’s that having an app available for installation at a URL opens up a whole new mode of operation for Dropserver:
👻👻👻💀💀💀 UNATTENDED APP UPGRADES 💀💀💀👻👻👻
Boo! Scary… 🫢
Since the base URL returns a listing of versions for an app, Dropserver could reload the URL periodically and install new versions of the app with no user interaction. This cascades further: unattended app upgrades lead to unattended appspace data migrations. Even more scary!
The big challenge with unattended app installs and data migrations is that it puts all the responsibility on the developer. It means doing things while the user is not present to be the ultimate guardian for what happens to their data. Naturally, we should never count on the end user to be the last-resort safety net. Users click “Yes” if they want the thing, so it should always be up to us to protect users as much as possible. But having a user click “yes” is a convenient cop-out for not doing everything we can to prevent a user getting into a bad situation.
Since apps and appspaces will eventually be upgraded without a user present, I need to do the work make it as safe as possible. Practically speaking, here are some ways I will try to do this:
Granular control
Granular control over automatic steps: user should be able to choose what they’re comfortable letting Dropserver do without them in attendance. Maybe fetching an app listing automatically is OK, and maybe installing the app unattended is fine. But migrating an appspace’s data? No way.
Guardrails
A system of warnings and good default guardrails. Since the user is not present to spot something “fishy” or that “looks weird”, Dropserver has to make those determinations. Some examples: if the license changes, if the app signature changes (not implemented yet), if the fetch URL redirects to a new, different domain, etc… would all cause the auto-upgrade to abort.
Usable Logging
What happened, and what was the outcome? A “log” of all operations, along with the results should be visible to the user and formatted in an easily understandable way. When operations are initiated by the user we can present the outcome of the op to them immediately and discard these results soon after. When things take place with the user absent, we have to stash all these operation results and make them available for perusal. This will not prevent problems of course, but can help the user what happened when something did go wrong.
Multi-step operations
In addition to all the above I want to lay the groundwork for a better UX when creating an Appspace. Currently, the user has to install an app, and then create an appspace for it. It’s really not good that these are two separate steps. In a future version I would really like for the app installation to be part of the appspace creation. The flow should be: drag an app URL into Dropserver, choose a subdomain, make a few selections for permissions if needed, and click “Create”.
This implies that the operation has to install the app and then create an appspace using the new app ID. Not a big problem but if you don’t think about it and plan it well, it’s a good way to end up with some serious spaghetti code.
So what code did I actually write?
After all that thinking I realized I needed to clean things up a bit in my appgetter code, so I refactored some of that to allow me to proceed more easily. Git commit b12d3c.
Then finally I wrote just enough code to be able to fetch and install an app from a URL. It’s incomplete but it’s functional as far as it goes. Git commit 35d6ae.
Currently I create the app website by hand. It’s utterly minimal, just enough to expose a link to a JSON and a few other files. I created bad sites too (missing manifest and package files) to check that error conditions are handled decently enough. I uploaded the sites to Netlify to allow me to test. You can see them here.
Note that part of the “apps from URLs” project is to make ds-dev generate a basic website capable of displaying app information to potential users as well as serve the right files to ds-host for the installation process.
SSRF protection
In the last few days of this month I started integrating an SSRF protection library to the fetch code.
I settled on Daenney’s SSRF library, instead of safeurl. This in spite of the fact that safeurl has 77 stars while Daenney’s has 1. I usually prefer to go with widely used libraries, so here is why:
- Neither is really widely used apparently, so it’s a tossup in that category anyways.
- Daenney’s SSRF is not a big library and I feel I understand the code well enough. I can patch it myself if need be.
- I didn’t like that safeurl disables IPv6 by default. It makes me less confident about the library for some reason. SSRF embraces IPv6 from the get go and from what I can tell the developer did their homework.
- I like that SSRF regenerates its list of default bad IP ranges from IANA sources twice a month. Neat!
- The blog post accompanying Daenney’s SSRF is informative and confidence-inspiring.
I’m adding ds-host config fields to specify IPs (and ranges) that should be reachable by the instance despite being on a private network. Work is ongoing.
Work on Leftovers and other apps
While I work on Dropserver proper, I also try to spend time on the apps that I write for Dropserver. One reason is that I actually use these apps daily, and they all have papercuts that I am motivated to fix.
I’m currently working on the Leftovers app. I would like it to be in good shape for when ds-dev can create app distribution websites. I’ll use Leftovers as the primary example of that.
What’s Next?
I’ll be plowing ahead with apps from URLs. I’d like to get to the point where apps and appspaces get upgraded when a new app version is available (if user so desires, of course). But I may not be able to make app installation part of the appspace creation operation (see “Multi-Step Operations” above). I need to draw the line somewhere.
By the end of November I hope to have ds-dev generate app distribution sites too. We’ll see.
