Dropserver Progress - September 2024
This is the progress report for Dropserver for September 2024. Here is last month’s report.
Going Off On a Tangent from Tailscale Integration
Tailscale integration is progressing. I am able to read information about the network, such as connection status, peers, etc… This data is dynamic and changes when the tailnet admin shares a node with someone, or a user connects a new device, etc… Some of this data has implications for the ds-host appspace owner. In particular, if an appspace node is shared with someone new, that person should be considered a user of the appspace, and that should be reflected on the frontend.
In 2024 you should not need to reload your interface if something changes externally. The UI the user is looking it should update dynamically to reflect the current reality. As such I am going to have to pipe some of that dynamic data from the tailnet to the frontend. This isn’t a big new thing for ds-host, I already have a rudimentary event system on the backend, and I use my self-developed protocol called “Twine” that can carry these events to the frontend. But…
Twine is overkill for frontend events
Twine is meant to offer advanced two-way communication between two running programs over a network. With Twine you can create subscriptions to a specific event and then unsubscribe. Twine is powerful, and I think I have some good ideas in there, but I’m realizing that frontends are best served with simple event streams. You don’t need powerful two-way communication for a typical frontend. Better to just get a firehose event stream.
Furthermore, Twine is under-developed: there are painful design mistakes, it’s cumbersome to use and debugging issues is unpleasant. I think it could be quite good but I would need to put a lot more effort into the protocol and the implementations for it to be a nice thing to use. Right now it’s just not, and I have too much on my plate to give it the attention it needs.
Send events to the ds-host fronted using Server-Sent Events
I decided that before sending events from the tailnet to the frontend, I should make sure I’m happy with how I do this. I don’t want to write more Twine code if I know I want to eliminate Twine. I don’t want to have two systems either: one Twine-based and one SSE-based. So I bit the bullet and decided to remove Twine from ds-host, at least for frontend events. (The communications between the sandbox and ds-host take place over Twine. In this case the two-way nature of it is useful. For now.)
To simplify things as much as possible, I decided that there would be a single SSE endpoint that would send events for all things that the logged in user might potentially be interested in. That doesn’t mean the user will see all events. The frontend may ignore some, or simply make a change to reactive data that will update in the UI if it is currently rendered. In other cases, like if the application they just started installing has completed, they might get a toast or something. The point is the frontend doesn’t subscribe to specific events. It’s a firehose. But the number of events should be small enough that it won’t matter.
Making all these changes forced me to go through all my backend event code. I refactored a number of areas. Events on the backend are more consistent now. They use the same generic base structs which have decent test coverage. I also found that some older parts of the code used events differently, so I made things more uniform throughout.
Commits 84b5dc8, df43dfa, 1a5eb34.
I’m amazed at how little JavaScript it takes to handle events on the frontend. I cut out a lot of code by dropping Twine. At some point I want to rebuild the frontend of ds-host to be plain server-rendered HTML pages, with some JS to make the dynamic parts live, and this lean events system is a better place to start.
One downside of SSEs is they take up one connection under http/1.1. But under http/2 that’s not an issue. ds-host should usually be served over h2, and even if not, it’s not a very request-hungry app, so it shouldn’t matter.
Deno 2 Drops
Deno 2 finally dropped! At least in release candidate form. So it was time to start testing and adapting my code to make sure everything would continue to work.
I made a number of changes to accommodate Deno 2. Most of the changes are minor, but existing apps need to be upgraded individually to work with Deno 2.
At the very least your app code should import version 0.2.2 of the dropserver_app library.
My vision for Dropserver is that apps should not need much maintenance, so asking devs to make changes to support Deno 2 bugs me. But it’s not easily avoidable right now. I’ll write more about Deno 2 and how Dropserver can better handle this kind of situation in a separate post.
Dropserver 0.13.3 Drops
On Oct 1 I tagged 0.13.3 of Dropserver which works with Deno 2 and includes the SSE changes mentioned above plus other commits that had accumulated lately.
There is no downside to upgrading to Dropserver 0.13.3, but I’d wait to upgrade your Deno install until you are sure all your apps support Deno 2.
Having said that it’s trivial to downgrade your Deno back to 1.x if things don’t work. (Trust me, I up/down-graded my Deno install a bunch of times over the last couple of weeks!)
What’s Next?
With the SSE work done, I’ll go back to the Tailscale integration work. The key thing is to make sense of the users from the peers list and merge them with appspace users. It’s going to be a bit messy
Tune in next month for the gruesome details.
