The State of Sandboxing in Dropserver

One of the bigger challenges of developing Dropserver has been to somehow make it safe to run the user’s application code. In this long-ago post I relayed how I tried a number of different approaches, all of them being too difficult to make work until Deno arrived. Naturally that was not the end of the story.

But first…

Why Is Sandboxing Important in Dropserver?

The goal of Dropserver is to make it possible for regular users to run server-side code of their choosing. Currently, for all internet users except for a very slim minority, touching anything “online” means interacting with a server over which they have no control. Dropserver aims to change that by giving regular people the ability to do internet things without losing all control.

Dropserver users are able to upload and run “apps", which are basically a pile of JS code written by a well-intentioned developer. While the “app” may include a frontend that will run in the browser, the real value of Dropserver is that the app can run server-side code.

An arrangement like that is rife for abuse if no guardrails are in place. That “well-intentioned” developer could actually be trying to gain control of the machine Dropserver runs on. Even if the developer is a goodie, compromised dependencies could seek to turn a nice app into a vector for an attack.

As a result Dropserver treats all app code as potentially malicious and gives it as little access as possible to protect the safety of users running apps, the admin hosting the instance, and everybody else on that instance.

My goal is that a Dropserver user should feel free to install apps just to kick the tires, even if they don’t know who the developer is and won’t be bothered to read through the code looking for vulnerabilities. Dropserver should feel safe. I think this is essential for the ecosystem to develop.

Deno is the Sandbox

Deno claims it is a secure runtime, and it’s likely fairly good at that. Through its use Dropserver can limit what an app can do, such as where on disk it can write and whether the app can make network requests.

However no software is without bugs, and even in the absence of bugs our current reality is that Spectre is a vulnerability that is pervasive across all running code. (Note there are some mitigations against Spectre attacks built into Deno: for example high-precision timers are behind a flag.)

In any case we should not assume Deno is and will remain airtight. For this reason a second layer of protection is essential. This is especially true since there is a good chance many Dropserver installations will not update Deno in time, and will therefore be vulnerable if any of their app code decides to take advantage of the vuln.

Bubblewrap is the Other Sandbox

Julia Evans aka “B0rk” has a neat writeup where she was looking at various sandboxing tools, and found Bubblewrap. Bubblewrap basically takes the challenging work of creating Linux namespaces for your sandbox in code and makes it accessible in a convenient command line utility.

Bubblewrap can set up Linux Kernel sandboxing features by specifying a few flags. It also helps make your sandbox useful by binding directories or files from the host system into the sandbox. Without Bubblewrap or another similar tool I’d be left with calling Linux system calls directly which would be more error-prone.

After some playing around, I feel so much better letting bwrap do this work rather than trying to code it myself.

It’s interesting to note that another app platform, Sandstorm (and now Tempest), roll their own C code to achieve a similar goal. There has been talk of leveraging Bubblewrap, but since they’ve already done the work and they trust their C code, they are sticking to it. More power to them (and more responsibility :))

Dropserver 0.10

In Dropserver v0.10.0 I added support for wrapping Deno instances in bwrap. Here are some notes on the current implementation:

• Deno is dynamically linked and therefore needs access to /etc and some lib directories to run. Not great. I’m hoping we’ll get a statically linked version of Deno some day. This is being talked about.
• I do not have an HTTP proxy for the sandbox yet so I have to give access to /run on certain systems so that it can do DNS lookups.
• Deno requires /proc dir. With --unshare-pid that’s not a big problem since the processes in /proc will only be bwrap and deno.
• Unfortunately I am not using --unshare-pid right now. It causes bwrap to give me the pid of the forked bwrap, not the deno command that I am running. I’ll spare you the details, but while this is something I can work around I decided it was time to ship.
• I realized that DENO_DIR, where deno caches imported modules, etc… can be a vector for abuse. It has to be read-write in the sandbox, so if someone breaks out of Deno, they can alter cached code files and therefore compromise other apps without even touching them. Yikes. For this reason each appspace gets its own DENO_DIR. It slows the initial start, but after that it’s no different.
• Since DENO_DIR is per-appspace, Deno makes many requests to the outside to obtain all the imported code the first time the appspace is run. For this reason in the current implementation the bwrap sandbox must have full net access.

As you can see there are a lot of caveats to Dropserver’s use of Bubblewrap for now. The good news is that there is a functional starting point that I can build on.

Roadmap

Here is how I see Dropserver’s sandbox evolving:

• If/when a statically linked or musl-compatible version of Deno is available, stop binding all the libs and /etc in the sandbox.
• A HTTP(S) proxy would make it possible to not bind the parts of the OS responsible for DNS lookups. (Such a proxy will happen alongside the feature that allows apps make outbound http requests.)
• I spent some time experimenting with Linux virtual networking, such as following this post, but unfortunately I could not come up with a combination of veth and whatever that did what I needed. (Some networking stuff requires privileges, which my use of bwrap does not have). However, creating a unix socket that is accessible from in and out of the sandbox is very straightforward so I expect to make the proxy accessible through that. Note that for now Deno doesn’t understand proxies as unix sockets, but that may change. Deno leverages Reqwest, which is considering this feature. Just have to be patient: the reward will be turning off network access entirely for the bwrap sandbox.
• Look deeper at user namespaces. I’m using --unshare-user-try, I need to look more at what happens when the user namespace does not get created.
• Dig deeper into Spectre to ensure I’m doing everything I can to mitigate.
• Seccomp. Bubblewrap has the ability to limit kernel calls via seccomp, but I have never tried that. This should make for a much tighter sandbox.
• There is a new set of tools that can help isolate a process called Landlock. I would look into that however I’ll probably have to wait until Bubblewrap implements it, which for now is not planned.
• Explore the possibility of putting Deno installation under ds-host's control. That way it could automatically upgrade to latest patched versions as they become available.

My hope is that eventually Dropserver’s Bubblewrap sandbox will be lean and effective. Having this second layer of security around Deno makes me much more comfortable with the prospect of eventually running a Dropserver service and letting others upload and run anything they want (scary!)

Note that the work of sandboxing is never done. Witness the work that Cloudflare puts into sandboxing its Workers.

Sandboxing is, of course, only part of the picture, but it’s a critical part and I’m glad to be moving forward with it.