Using toolbox containers as a persistent, stable and easy-to-deploy development environment for data analysis work

A bit of preamble

A while ago, I asked for some advice on the Fediverse about using containers for your development environment when doing interactive R/Quarto data analysis work. What triggered the question was a bit of frustration that rig, the software I prefer to use for managing my R installations on my machine, wasn't keeping pace super well with Fedora's releases. This was sufficiently inconvenient that it forced me to deal with the fact that I shouldn't really be relying on my standard day-to-day OS for my critical work tools. They should be kept isolated in a stable environment so that I can manage or change my OS without worrying about breaking my projects or workflow. Basically, I probably needed to learn more about containerised workflows.

As part of my Fediverse call for help, Jim Gardner took an interest in my use case and wrote up a generous piece demonstrating what building a working container for my particular workflow might look like. This post is, in a sense, a very late follow up to his post.

In order to contextualise why it's taken me so long to write up a response, I need to explain that by the time I had worked through Jim's example and considered whether something like that might be just what I was looking for, I had already stumbled into a temporary, hacky solution to my Fedora-rig problems that enabled me to keep working with minimal disruption. As it happens, that hacky solution has worked so incredibly well for the last year or so that I became convinced that it, in fact, was the solution I was looking for all along.

The solution I found which ended up being the answer to my problems? Toolbox containers.

In this post, I'll talk about what the toolbox approach to containerisation is and what my setup looks like. I'll talk about why I think this approach can be such a perfect fit if your circumstances and needs look like mine - and I'll talk about why it's probably not the right tool if they don't. I'll also, finally, return to the point of Jim Gardner's post as I talk about my attempt to build an image that serves as a "batteries included" command line environment for my development needs - Neovim, Quarto, R and more, all available out of the box.

What are toolbox containers?

The primary use case for toolbox containers is providing a standard modifiable command line environment on atomic or "immutable" operating systems. In such OSes, only the user's home directory is writable, and the system directories are administered as part of an overall system image which cannot be modified. Basically, your home directory is yours to do what you want with, but the system only changes by swapping the image for an updated image.

There's several advantages and drawbacks to the atomic OS paradigm, but a consequence is that traditional distro package management doesn't work, because that approach usually writes packages and configuration files into system directories. Toolboxes get around this problem by allowing users to create persistent containerised environments on their system that are "mutable", allowing you to install packages the traditional way - they just live in the system of the container rather than the host. To reduce friction, toolbox containers have some key properties: the user is carried through from the host system and has sudo access inside the container, and the user's home directory is mounted inside the container. Together, these features mean that being inside a toolbox environment on an atomic OS is largely indistinguishable from being at a command prompt in a traditional package-based OS.

Toolbox vs Toolbx vs Distrobox: a note on tools and terms

Now's a good time to state that the term I'm using here - "toolbox containers" - isn't ideal. That's because one of the major pieces of software implementing the idea is called Toolbx (I don't know why they dropped the third "o"; the shell command is spelled "toolbox".) This overlap could get confusing, but unless I spell it without that third "o", I'm talking about the general concept, regardless of implementation.

I'm sticking to the more general term because, while Toolbx is a great piece of software, it isn't the only tool in this space. In fact, if you're not a Fedora user, you're probably more likely to have encountered Distrobox, which implements and expands on Toolbx's core ideas in a more general framework (and does it all using shell scripts!). Unlike Toolbx, Distrobox doesn't have a hard dependency on systemd, making it more broadly compatible as well.

There's also a new tool that I'm keeping an eye on called Kapsule, which is a KDE project that is being developed primarily to enrich the new immutable KDE Linux distro. This one's particularly interesting because it's not built on top of Docker/podman, and is specifically built with persistence in mind. It also has some interesting dials, like letting you decide how much of the host filesystem gets mounted into the container. It seems like a super promising option, but it's very early days.

Personally, I used Toolbx for probably around a year as I was a long time Fedora daily driver, and Toolbx is standard Fedora software. In fact, when I first drafted this post, I was still using Toolbx. However, a few months ago I picked up an open box ThinkPad for a good price (considering the... you know, everything, at the moment) and decided to use setting this machine up as an opportunity to change my daily driver to Void Linux. It's a great distribution, but it doesn't use systemd, which means I needed to leave Toolbx behind. Thankfully, it seems that any image that works as a Toolbx container will also work as a Distrobox container, so it was a very smooth switch.

Everything in this post should apply just as well to either Toolbx or Distrobox, so go with whatever's more appealing to you.

What makes this a good fit for data analysis work?

The primary use case is atomic OSes, but it's not the only use case. Both Toolbx and Distrobox list some examples of other use cases on their websites, and they can be summed up as "you want to mix and match the characteristics of your host OS and a command line environment". Funnily enough, Distrobox highlights the specific case of mixing a stable base OS with a bleeding-edge dev environment as one use case. I think what makes these tools so attractive for data science work is exactly the opposite case: run the latest and greatest software on your machine, while keeping your work setup protected in a stable development environment.

That's particularly important for those of us who work with data because we spend a lot of our time in our development environment, but we generally need to prioritise stability and continuity.

One example of this is for the sake of reproducibility. If my code produced one result last month, I probably want the same code to produce the same result today, regardless of what updates I've installed on my base system in the meantime.

Another example is making sure things "just work". Breaking changes can have a hugely disruptive impact on work and productivity, so I'm much more deliberate about updating workflow-critical tools than I am about all other software that I run. Maybe one of the pieces of software in my workflow has some cool new features, but I'll stay on a known-working version until I'm ready to deal with any friction that might arise from upgrading. I don't want these tools to be handled by my main distro package manager, or to fall too out of sync with any system-level dependencies that are handled by the package manager. Toolboxes provide a great framework for isolating the work tools, giving me more agency over what I update and when.

In setting out why this approach works so well for me, I think it's important to put some parameters on what I'm really talking about here.

I think using toolbox containers can be a great fit when:

• You do your work on a mixed-purpose machine - i.e. the computer that you work on is also a computer you use for other things.
• You want to replace working on your host OS with working in a controlled environment, rather than maintaining a mix of both workflows.
• You do a lot of exploratory data analysis work - dipping in and out of different projects that can share an environment (e.g. libraries, maybe even some data files), not necessarily wanting to set up each thing that you work on as a separate self-contained environment.
• Your work files live somewhere inside your home folder all the time, and are either synced to cloud storage or a source hosting repository.

On the other hand, it's probably not a good fit for you if:

• You work mostly on remote machines, and want your containers to be entirely self-contained so that they can be one-click deployed on a new machine without needing to configure anything on the host.
• You prefer to isolate projects from each other, only making the files and libraries of each project available inside its workspace.
• Your reason for using containers is motivated purely by the goal of reproducibility.

In these cases, a more traditional container-based workflow probably makes more sense than the approach offered by tools like Toolbx.

The pros and cons of a semi-containerised life

Regardless of whether any of the above scenarios apply to you, I think I can boil down the approach to the following lists of pros and cons.

Some advantages of the toolbox model:

• You're free to upgrade and tinker with your host OS as much as you want, knowing nothing in the container will break.
• You're free to choose your daily driver OS without worrying about whether it's a good fit for your work needs. If you want to run an Arch-based gaming distro on your computer but you know your R life will be much easier if you run something Debian-based because critical tools in your field don't support Arch, you can just use a Debian-based image as a base for a toolbox container, no problem.
• You can very quickly deploy an environment that has all your workflow tools, allowing you to get up and running faster on a new machine.
• Your dotfiles will be picked up inside the container.
• Your user-level R package library can be picked up inside each toolbox container (though this could end up being a bit of a headache, depending on your setup).
• Any software that you make available through your host OS's home directory (e.g. binaries that live in ~/.local/bin) will be available inside the container, which can be very useful if there is software you want to share between host and container without installing it twice.

And some disadvantages:

• The tight integration of the host system means that you don't get as many guardrails as you would in a traditional container, and you need to be as careful as you would when operating on your host system's command line.
• Being inside a container can make some tasks a little bit more complicated - for example, R's standard way of opening a web browser invokes xdg-open, but we need to replace it with a call to something like flatpak-xdg-open which allows the container to open the process on the host.
• Because the container will get its home directory from the host system, it means that it can't really be pre-loaded with things like dotfiles, as it needs to have no home directory when it's created. This means that a Toolbx image isn't quite as "batteries included" as a regular container image can be, because you'll need to bring your own dotfiles, project files, etc. This isn't a huge issue because you only need to source those files once, but it is a drawback.
• Transitioning from a host-based workflow to a toolbox-style workflow on the same machine may not be entirely smooth. It could be the case, for example, that R packages that were installed on your host system will start to throw errors inside the container because the container's system setup (presence of dependencies, location of library files etc) is different from the host's.

Rolling your own toolbox

It's pretty easy to get up and running, as the Toolbx project also publishes a number of base images that can be used as starting points for containers. For example,using Toolbx, toolbox create --distro fedora --release f42 f42box creates a toolbox called "f42box" based on Fedora 42. Alternatively, toolbox create --image quay.io/toolbx-images/debian-toolbox:13 creates a toolbox called "debian-toolbox-13" from the community Debian 13 base image.

Then, you can enter the toolbox you created with toolbox enter *toolbox-name* and start to customise it to your liking.

This is the approach I used for about a year or so - I started with a Fedora base image, and I began installing the stuff I use inside it, as I would on my host machine.

However, when setting up the new ThinkPad I wanted to take the opportunity to streamline the setup process for this new machine, and the future machines that might follow it. I knew I wanted to use a similar toolbox-based setup to the one I had been using, but I wanted to be able to start from a base image that was already tailored to my workflow and needs, instead of starting from zero. In other words, it was time to build my own custom toolbox image.

The process of building a toolbox image is the same as building any other OCI container image, and thankfully, Jim Gardner wrote up a really handy guide on that a little while ago. The only difference for a toolbox image is that it just needs to conform to the specific requirements that Toolbx has - the most important and tricky of which is that it needs to have an empty home directory. The easiest way to ensure you're Toolbx-compatible is to start with one of the standard Toolbx images as a base, since they're already tailored for that purpose. Let's start there.

Walking through my setup

I built my custom image from a Containerfile, because I like doing things in a declarative way where possible. Here it is in full:

FROM registry.fedoraproject.org/fedora-toolbox:43

RUN dnf --assumeyes copr enable dejan/lazygit
RUN dnf --assumeyes copr enable atim/starship
RUN dnf install --assumeyes fastfetch flatpak-xdg-utils fuse3 lazygit ripgrep starship stow uv wl-clipboard

RUN <<EOF
curl -LO https://github.com/neovim/neovim/releases/download/v0.11.7/nvim-linux-x86_64.appimage
chmod u+x nvim-linux-x86_64.appimage
mv ./nvim-linux-x86_64.appimage /usr/local/bin/nvim
chmod go+rx /usr/local/bin/nvim
EOF

RUN dnf install --assumeyes https://github.com/r-lib/rig/releases/download/latest/r-rig-latest-1.$(arch).rpm
RUN rig add --without-p3m 4.4
RUN rig add --without-p3m 4.5
RUN rig add --without-p3m release
RUN rig system add-pak --all

RUN <<EOF
curl -Lo /tmp/arf.tar.xz https://github.com/eitsupi/arf/releases/latest/download/arf-console-x86_64-unknown-linux-gnu.tar.xz
tar -C /tmp -xf /tmp/arf.tar.xz
mv /tmp/arf-console-x86_64-unknown-linux-gnu/arf /usr/local/bin/arf
chown root:root /usr/local/bin/arf
rm -r /tmp/arf-console-x86_64-unknown-linux-gnu /tmp/arf.tar.xz
/usr/local/bin/arf --version
EOF

ENV QUARTO_VERSION="1.10.18"
RUN <<EOF
mkdir -p /opt/quarto/${QUARTO_VERSION}
curl -o /tmp/quarto.tar.gz -L https://github.com/quarto-dev/quarto-cli/releases/download/v${QUARTO_VERSION}/quarto-${QUARTO_VERSION}-linux-amd64.tar.gz
tar -C /opt/quarto/${QUARTO_VERSION} -xzvf /tmp/quarto.tar.gz --strip-components=1
rm /tmp/quarto.tar.gz
ln -s /opt/quarto/${QUARTO_VERSION}/bin/quarto /usr/local/bin/quarto
/usr/local/bin/quarto check
EOF

ENV RV_VERSION="0.22.2"
RUN <<EOF
mkdir -p /opt/rv/${RV_VERSION}
curl -o /tmp/rv.tar.gz -L https://github.com/A2-ai/rv/releases/download/v${RV_VERSION}/rv-v${RV_VERSION}-x86_64-unknown-linux-gnu.tar.gz
tar -C /opt/rv/${RV_VERSION}/ -xzvf /tmp/rv.tar.gz
rm /tmp/rv.tar.gz
chmod +x /opt/rv/${RV_VERSION}/rv
ln -s /opt/rv/${RV_VERSION}/rv /usr/local/bin/rv
/usr/local/bin/rv --version
EOF

RUN dnf clean all

It should be pretty self-explanatory if you've seen a Containerfile before (or if you read Jim Gardner's post!), but I'll explain my choices a little bit.

I originally explored using a Debian base image for my development environment, as I wondered whether Fedora's release cadence might be a little too fast for the purpose at hand, and Debian is usually the go-to for stability. However, after years of Fedora usage, I was pretty shocked at how out of date the software in the Debian repos (not just in stable!) was compared to Fedora. So, in the end, I opted for Fedora 43 as a starting point, which should be perfectly fine for the foreseeable future, and requires less tinkering out of the box.

The bulk of the Containerfile is basically just installing the software I want to be available out of the box. I use rig for R installation management, and rig provides an RPM package for Fedora 43, making installation a breeze. Using rig, I make sure I have R 4.4.x and 4.5.x available in the container, the two versions I have projects with dependencies on. The rest is fairly straightforward if you know the shape of my workflow from previous posts. Neovim for editing, lazygit for lazy git, Quarto for computational notebooks, uv and rv for managing Python and R projects. I chose to use the new kid on the block arf as the R REPL, since it comes as a single binary that's a bit easier to install than radian, which is a Python package (and is also no longer being actively maintained, unfortunately). After using arf for a few months, this was a good decision - it's great software.

Normally, I'd install install a lot of these tools on my system by symlinking the binaries into ~/.local/bin, but since we need to stay out of the home directory, I instead target /usr/local/bin in the Containerfile.

That's it, really. Not much else needs to be done. Toolbx or Distrobox handle all the configuration of entrypoints and mounts etc. Environment variables can be picked up from your dotfiles later. As long as I clone my Neovim config into ~/.config/nvim at some point (note that I could do this from the host), then when I run nvim for the first time inside the toolbox, it will configure itself how I like and get all my plugins installed.

If you really want to pre-configure some of the software, and the software allows config files to be placed in a system-level config file location, you could handle putting your config files there within the Containerfile. Just bear in mind that if you also have a config file in the user-level config file location, that will usually take precedence.

I suppose there's also the question of pre-installing R packages somewhere outside of the home directory and making sure that library is added to your R profile. But I decided I'd rather just do a supervised installation of R packages after setup.

Deployment

You can build the image from the Containerfile by running:

podman build --squash --tag localhost/f43custom .

from the directory with the Containerfile in it.

Then, if everything builds smoothly, you create your Toolbx with

toolbox create --image f43custom workbox

Or, in my case (the --unshare-process flag is needed on a systemd-free system):

distrobox create --image f43custom --name workbox --unshare-process

and enter it with toolbox enter workbox / distrobox enter workbox. I also have an alias set up so that I can run distrobox enter workbox && nvim in a brief command.

Takeaways

I should point out that the Containerfile above is still only an early attempt at this, and I'll probably continue to tinker around the edges of my setup, as usual. I am interested in whether I can get the size of the uncompressed image down, though I don't have the spare time to go down that rabbit hole just yet. For now, I'm pretty happy just to have something that works as intended.

For my use case, the goal is not to constantly update the Containerfile and rebuild the container with the latest and greatest versions of the software. When a new version of arf or rv or Quarto is released and I want to move to it, I simply download it, enter the toolbox, and move the new version to /usr/local/bin etc in the toolbox. No need to rebuild or anything. In general, when a new tool altogether is added to the container or a major change is made, then it's time to update the Containerfile to reflect that. It's about keeping the Containerfile in sync with the state of the container, so that I can feel ready to rebuild it quickly if I needed to.

Obviously, this kind of thing is always going to end up being pretty specific to the needs and preferences of the individual user. But if it turns out that I'm not the only person that finds this approach useful, I wonder whether having a few curated starting images available would be a nice resource for the R/data science/computational social science community. The rocker project has been a great resource for people, and I note that there is already some pretty good representation in the devcontainer spec space for features from the data science world. Maybe a goal could be to build a stock of tried and tested Toolbx-compatible images that are tailored to data science workflows, and push these out to a container registry for others to use as a starting point.

In the meantime, I've pushed mine to Docker Hub in case anyone wants to try it out without having to build the image from the Containerfile. The repo has the Fedora 43-based image, which is tagged f43, and it has an equivalent image which is Debian-based, tagged debian13.

In any case, let me know if you found this post useful, or if you know of other tools that complement the kind of approach I've described here.