You don't need CI to deploy a static website

2026-09-18T00:00:00+00:00

I had a rare surplus of energy this past week and I decided to divert it towards seeing if there was any maintenance that needed to be done on my public projects. This ended up with me blowing the cobwebs off {catchup}</a>, an R package I use all the time in my personal workflow but rarely hack on. I realised that the package might have a bit more appeal to somebody other than me if it had a nice-to-look-at documentation website rather than just the README of a rarely updated source repo.

A day or two later, after a bit of reading and some trial and error, I had managed to deploy the site at my own domain.

And the best part is that it used no GitHub and no CI.

I was so won over by how much better the setup was than the one I had been using for my main website (i.e. this one) for years, that I swiftly switched it over to the same deployment system.

This post might be a bit of a departure from what I usually write about because it's not as focused on data and research. But I think there are a lot of people like me with no software development background, who have ended up in the world of programming and version control and code hosting repositories mostly to support their research activities. Maybe this post will be helpful in providing an example of a pretty easy alternative to the vast amount of material assuming you're happy to adopt a setup based on GitHub and/or CI.

Why drop CI?</h2>
I don't think I need to rehearse the reasons why you would want to avoid GitHub at this point - you either get it or you don't. But it might be less obvious why I'm apparently lumping CI into the same category, so I guess I should expand on that.
The main issue with CI is that it's resource-intensive. That's not as much of a problem for big software projects, where the cost is worth it to ensure that the test suite or the build pipeline is being run at certain points in the development process. But tools like GitHub Actions have abstracted away the costs and realities of CI to the extent that it is overused on even tiny projects, a practice which has been super normalised in the data science world.
It's not that CI is inherently bad, it's just not always the approriate tool for the job in my opinion, regardless of how frictionless it is.
CI and static site generators</h3>
Let's use static site generators (SSGs) as our example. Typically, for a website built with an SSG, you have a source directory that follows a set structure, containing the theme, templates, configuration files and content for the website. It might look something like this:
`content/ static/ templates/ theme/ config.toml </code></pre> Then you run the SSG program's build command from the root of that directory to generate a new directory containing the static website files, which is what will actually be served:`
`build/ (<- new, contains the static site) content/ static/ templates/ theme/ config.toml </code></pre> There are a few different options for how you could deploy the site. The simplest version doesn't involve any version control at all - you can just take that the contents of that build/</code> directory and drop them anywhere that offers web hosting. But the two main options I want to consider are the following:`
`Option 1: Put the source files in version control, and build the site via CI. (Generally, this will then integrate with something like Netlify or GitHub Pages to serve the built site.)</li>`Option 2: Put both the source files and the locally built site in version control. You can use webhooks to serve the built site, updating on each push event.</li> </ul> In the first option, you don't build the site locally, you set up a CI workflow that builds the site whenever you push the source. So, every time you make a public-facing change to the website, you spin up a virtual machine just to run a simple build command that would take about a second to run on your laptop (where you're already working on the source files), and then throw that machine away. Is this necessary? This seems like using a chainsaw to open a can of beans to me. Option 2 gets to exactly the same end state, without any of the pointless waste. It's also more portable and less tied to specific platforms and services. I think it makes way more sense, but it's much less well documented. For example, I built the {catchup} website with {pkgdown}</a>, which is a really fantastic package. However, {pkgdown} only really targets a GitHub Actions-based workflow for deployment of the websites it produces, and it makes very little effort to explain other strategies for getting your website online. I guess that's part of the gap that I'm trying to close here by writing up what I did and why I find it so much more sensible. To explain that, there's two parts to the setup that need to be outlined: git-pages, and git worktrees. Deploying via git-pages with Codeberg</h2> The first tool that really helps make this easy is git-pages</a>, a relatively new tool that has been developed to underpin Codeberg's Pages functionality. It's a git-based static site server - it takes a repo and serves a website from it. It provides the basis for forges like Codeberg to offer something like what GitHub Pages can offer - a way of saying "if you have your static site files in a repo hosted here, we have a way of serving the site over the web". I won't go too much into the details of how to set up a repo to take advantage of it because honestly I just followed the excellent documentation</a>. The important point to note here is that Codeberg's implementation expects that, for a repository-based website, your site files live in the root of a branch called pages</code>. So, if you recall the example directory strucure from earlier, that would mean taking everything inside build/</code>, .gitignore</code>-ing it, and checking it out at a separate branch pages</code>, while the source continues to be in a separate branch (e.g. main</code>). There are various ways to arrive at this situation, but the one that makes this whole process a breeze is git worktrees, which I'll get to in a second. Once you have the site files in your public pages</code> branch, you can set up a Forgejo webhook following the instructions in the Codeberg docs that will re-serve your site every time there is a new push event to the branch. How git worktrees help</h2> Perhaps you're already using git worktrees</a>, but I'll admit that I hadn't yet found a use for them until discovering how perfectly they suit this branch-based deployment setup. Git worktrees allow you to have a subdirectory within a repository act like a checked out branch of the repository, so that when you navigate to the subdirectory you checkout that branch. This is the perfect mechanism for declaring that the build/</code> subdirectory of the repo should map to the pages</code> branch, as this post</a> points out. When the SSG builds the static site directory, it puts the files where they need to be - in the root of the pages</code> branch. One thing to be careful of: the way that git keeps track of the worktree is via a .git file in the subdirectory, and some static site generators completely delete and rebuild the subdirectory each time they call their build command. In that case, you need to temporarily move the .git file somewhere safe and put it back in place after and everything will work fine. Thankfully, my SSG of choice, Zola</a> has a configuration option to preserve dotfiles in the build directory precisely for this use case. So, if your webhook and your worktree are set up correctly, after updating your website's source, you can deploy your website changes just by doing something like the following: myfavssg build cd build git add -A git commit -m "fix typos in latest post" git push -u origin pages </code></pre> And the site will be updated. To sum up</h2> Whether it's a {pkgdown} site, or a site built by something like Zola or Hugo - if it generates a subdirectory with static site files, you can set it up as a worktree and point it to a pages</code> branch that gets served via a webhook with Codeberg Pages. Switching this site to use that deployment system meant I no longer had any ties to Netlify, and I could take back my DNS management from them for my domain, which is a bonus. It also enabled me to reach repository zero on GitHub. Let me know if there's anything I've not been clear on here, or if you have any other tips on how to make this kind of thing work smoothly that I might have missed.

Using toolbox containers as a persistent, stable and easy-to-deploy development environment for data analysis work

2026-07-27T00:00:00+00:00

A bit of preamble</h2>
A while ago, I asked for some advice on the Fediverse about using containers for your development environment when doing interactive R/Quarto data analysis work. What triggered the question was a bit of frustration that rig</a>, the software I prefer to use for managing my R installations on my machine, wasn't keeping pace super well with Fedora's releases. This was sufficiently inconvenient that it forced me to deal with the fact that I shouldn't really be relying on my standard day-to-day OS for my critical work tools. They should be kept isolated in a stable environment so that I can manage or change my OS without worrying about breaking my projects or workflow. Basically, I probably needed to learn more about containerised workflows.
As part of my Fediverse call for help, Jim Gardner took an interest in my use case and wrote up a generous piece</a> demonstrating what building a working container for my particular workflow might look like. This post is, in a sense, a very late follow up to his post.
In order to contextualise why it's taken me so long to write up a response, I need to explain that by the time I had worked through Jim's example and considered whether something like that might be just what I was looking for, I had already stumbled into a temporary, hacky solution to my Fedora-rig problems that enabled me to keep working with minimal disruption. As it happens, that hacky solution has worked so incredibly well for the last year or so that I became convinced that it, in fact, was the solution I was looking for all along.
The solution I found which ended up being the answer to my problems? Toolbox containers.
In this post, I'll talk about what the toolbox approach to containerisation is and what my setup looks like. I'll talk about why I think this approach can be such a perfect fit if your circumstances and needs look like mine - and I'll talk about why it's probably not the right tool if they don't. I'll also, finally, return to the point of Jim Gardner's post as I talk about my attempt to build an image that serves as a "batteries included" command line environment for my development needs - Neovim, Quarto, R and more, all available out of the box.

What are toolbox containers?</h2>
The primary use case for toolbox containers is providing a standard modifiable command line environment on atomic or "immutable" operating systems. In such OSes, only the user's home directory is writable, and the system directories are administered as part of an overall system image which cannot be modified. Basically, your home directory is yours to do what you want with, but the system only changes by swapping the image for an updated image.
There's several advantages and drawbacks to the atomic OS paradigm, but a consequence is that traditional distro package management doesn't work, because that approach usually writes packages and configuration files into system directories. Toolboxes get around this problem by allowing users to create persistent containerised environments on their system that are "mutable", allowing you to install packages the traditional way - they just live in the system of the container rather than the host. To reduce friction, toolbox containers have some key properties: the user is carried through from the host system and has sudo access inside the container, and the user's home directory is mounted inside the container. Together, these features mean that being inside a toolbox environment on an atomic OS is largely indistinguishable from being at a command prompt in a traditional package-based OS.
Toolbox vs Toolbx vs Distrobox: a note on tools and terms</h3>
Now's a good time to state that the term I'm using here - "toolbox containers" - isn't ideal. That's because one of the major pieces of software implementing the idea is called Toolbx</a> (I don't know why they dropped the third "o"; the shell command is spelled "toolbox".) This overlap could get confusing, but unless I spell it without that third "o", I'm talking about the general concept, regardless of implementation.
I'm sticking to the more general term because, while Toolbx is a great piece of software, it isn't the only tool in this space. In fact, if you're not a Fedora user, you're probably more likely to have encountered Distrobox</a>, which implements and expands on Toolbx's core ideas in a more general framework (and does it all using shell scripts!). Unlike Toolbx, Distrobox doesn't have a hard dependency on `systemd</code>, making it more broadly compatible as well.`
`There's also a new tool that I'm keeping an eye on called` `Kapsule</a>, which is a KDE project that is being developed primarily to enrich the new immutable KDE Linux</a> distro. This one's particularly interesting because it's not built on top of Docker/podman, and is specifically built with persistence in mind. It also has some interesting dials, like letting you decide how much of the host filesystem gets mounted into the container. It seems like a super promising option, but it's very early days.`
Personally, I used Toolbx for probably around a year as I was a long time Fedora daily driver, and Toolbx is standard Fedora software. In fact, when I first drafted this post, I was still using Toolbx. However, a few months ago I picked up an open box ThinkPad for a good price (considering the... you know, everything, at the moment) and decided to use setting this machine up as an opportunity to change my daily driver to Void Linux. It's a great distribution, but it doesn't use systemd</code>, which means I needed to leave Toolbx behind. Thankfully, it seems that any image that works as a Toolbx container will also work as a Distrobox container, so it was a very smooth switch.
`Everything in this post should apply just as well to either Toolbx or Distrobox, so go with whatever's more appealing to you.`
What makes this a good fit for data analysis work?</h3> The primary use case is atomic OSes, but it's not the only use case. Both Toolbx</a> and Distrobox</a> list some examples of other use cases on their websites, and they can be summed up as "you want to mix and match the characteristics of your host OS and a command line environment". Funnily enough, Distrobox highlights the specific case of mixing a stable base OS with a bleeding-edge dev environment as one use case. I think what makes these tools so attractive for data science work is exactly the opposite case: run the latest and greatest software on your machine, while keeping your work setup protected in a stable development environment. That's particularly important for those of us who work with data because we spend a lot of our time in our development environment, but we generally need to prioritise stability and continuity. One example of this is for the sake of reproducibility. If my code produced one result last month, I probably want the same code to produce the same result today, regardless of what updates I've installed on my base system in the meantime. Another example is making sure things "just work". Breaking changes can have a hugely disruptive impact on work and productivity, so I'm much more deliberate about updating workflow-critical tools than I am about all other software that I run. Maybe one of the pieces of software in my workflow has some cool new features, but I'll stay on a known-working version until I'm ready to deal with any friction that might arise from upgrading. I don't want these tools to be handled by my main distro package manager, or to fall too out of sync with any system-level dependencies that are handled by the package manager. Toolboxes provide a great framework for isolating the work tools, giving me more agency over what I update and when. In setting out why this approach works so well for me, I think it's important to put some parameters on what I'm really talking about here. I think using toolbox containers can be a great fit when: You do your work on a mixed-purpose machine - i.e. the computer that you work on is also a computer you use for other things.</li> You want to replace working on your host OS with working in a controlled environment, rather than maintaining a mix of both workflows.</li> You do a lot of exploratory data analysis work - dipping in and out of different projects that can share an environment (e.g. libraries, maybe even some data files), not necessarily wanting to set up each thing that you work on as a separate self-contained environment.</li> Your work files live somewhere inside your home folder all the time, and are either synced to cloud storage or a source hosting repository.</li> </ul> On the other hand, it's probably not a good fit for you if: You work mostly on remote machines, and want your containers to be entirely self-contained so that they can be one-click deployed on a new machine without needing to configure anything on the host.</li> You prefer to isolate projects from each other, only making the files and libraries of each project available inside its workspace.</li> Your reason for using containers is motivated purely by the goal of reproducibility.</li> </ul> In these cases, a more traditional container-based workflow probably makes more sense than the approach offered by tools like Toolbx. The pros and cons of a semi-containerised life</h3> Regardless of whether any of the above scenarios apply to you, I think I can boil down the approach to the following lists of pros and cons. Some advantages of the toolbox model: You're free to upgrade and tinker with your host OS as much as you want, knowing nothing in the container will break.</li> You're free to choose your daily driver OS without worrying about whether it's a good fit for your work needs. If you want to run an Arch-based gaming distro on your computer but you know your R life will be much easier if you run something Debian-based because critical tools in your field don't support Arch, you can just use a Debian-based image as a base for a toolbox container, no problem.</li> You can very quickly deploy an environment that has all your workflow tools, allowing you to get up and running faster on a new machine.</li> Your dotfiles will be picked up inside the container.</li> Your user-level R package library can be picked up inside each toolbox container (though this could end up being a bit of a headache, depending on your setup).</li> Any software that you make available through your host OS's home directory (e.g. binaries that live in ~/.local/bin</code>) will be available inside the container, which can be very useful if there is software you want to share between host and container without installing it twice.</li> </ul> And some disadvantages: The tight integration of the host system means that you don't get as many guardrails as you would in a traditional container, and you need to be as careful as you would when operating on your host system's command line.</li> Being inside a container can make some tasks a little bit more complicated - for example, R's standard way of opening a web browser invokes xdg-open</code>, but we need to replace it with a call to something like flatpak-xdg-open</code> which allows the container to open the process on the host.</li> Because the container will get its home directory from the host system, it means that it can't really be pre-loaded with things like dotfiles, as it needs to have no home directory when it's created. This means that a Toolbx image isn't quite as "batteries included" as a regular container image can be, because you'll need to bring your own dotfiles, project files, etc. This isn't a huge issue because you only need to source those files once, but it is a drawback.</li> Transitioning from a host-based workflow to a toolbox-style workflow on the same machine may not be entirely smooth. It could be the case, for example, that R packages that were installed on your host system will start to throw errors inside the container because the container's system setup (presence of dependencies, location of library files etc) is different from the host's.</li> </ul> Rolling your own toolbox</h2> It's pretty easy to get up and running, as the Toolbx project also publishes a number of base images that can be used as starting points for containers. For example,using Toolbx, toolbox create --distro fedora --release f42 f42box</code> creates a toolbox called "f42box" based on Fedora 42. Alternatively, toolbox create --image quay.io/toolbx-images/debian-toolbox:13</code> creates a toolbox called "debian-toolbox-13" from the community Debian 13 base image</a>. Then, you can enter the toolbox you created with toolbox enter toolbox-name</code> and start to customise it to your liking. This is the approach I used for about a year or so - I started with a Fedora base image, and I began installing the stuff I use inside it, as I would on my host machine. However, when setting up the new ThinkPad I wanted to take the opportunity to streamline the setup process for this new machine, and the future machines that might follow it. I knew I wanted to use a similar toolbox-based setup to the one I had been using, but I wanted to be able to start from a base image that was already tailored to my workflow and needs, instead of starting from zero. In other words, it was time to build my own custom toolbox image. The process of building a toolbox image is the same as building any other OCI container image, and thankfully, Jim Gardner wrote up a really handy guide</a> on that a little while ago. The only difference for a toolbox image is that it just needs to conform to the specific requirements</a> that Toolbx has - the most important and tricky of which is that it needs to have an empty home directory. The easiest way to ensure you're Toolbx-compatible is to start with one of the standard Toolbx images as a base, since they're already tailored for that purpose. Let's start there. Walking through my setup</h3> I built my custom image from a Containerfile</a>, because I like doing things in a declarative way where possible. Here it is in full: FROM registry.fedoraproject.org/fedora-toolbox:43 RUN dnf --assumeyes copr enable dejan/lazygit RUN dnf --assumeyes copr enable atim/starship RUN dnf install --assumeyes fastfetch flatpak-xdg-utils fuse3 lazygit ripgrep starship stow uv wl-clipboard RUN <<EOF curl -LO https://github.com/neovim/neovim/releases/download/v0.11.7/nvim-linux-x86_64.appimage chmod u+x nvim-linux-x86_64.appimage mv ./nvim-linux-x86_64.appimage /usr/local/bin/nvim chmod go+rx /usr/local/bin/nvim EOF RUN dnf install --assumeyes https://github.com/r-lib/rig/releases/download/latest/r-rig-latest-1.$(arch).rpm RUN rig add --without-p3m 4.4 RUN rig add --without-p3m 4.5 RUN rig add --without-p3m release RUN rig system add-pak --all RUN <<EOF curl -Lo /tmp/arf.tar.xz https://github.com/eitsupi/arf/releases/latest/download/arf-console-x86_64-unknown-linux-gnu.tar.xz tar -C /tmp -xf /tmp/arf.tar.xz mv /tmp/arf-console-x86_64-unknown-linux-gnu/arf /usr/local/bin/arf chown root:root /usr/local/bin/arf rm -r /tmp/arf-console-x86_64-unknown-linux-gnu /tmp/arf.tar.xz /usr/local/bin/arf --version EOF ENV QUARTO_VERSION="1.10.18" RUN <<EOF mkdir -p /opt/quarto/${QUARTO_VERSION} curl -o /tmp/quarto.tar.gz -L https://github.com/quarto-dev/quarto-cli/releases/download/v${QUARTO_VERSION}/quarto-${QUARTO_VERSION}-linux-amd64.tar.gz tar -C /opt/quarto/${QUARTO_VERSION} -xzvf /tmp/quarto.tar.gz --strip-components=1 rm /tmp/quarto.tar.gz ln -s /opt/quarto/${QUARTO_VERSION}/bin/quarto /usr/local/bin/quarto /usr/local/bin/quarto check EOF ENV RV_VERSION="0.22.2" RUN <<EOF mkdir -p /opt/rv/${RV_VERSION} curl -o /tmp/rv.tar.gz -L https://github.com/A2-ai/rv/releases/download/v${RV_VERSION}/rv-v${RV_VERSION}-x86_64-unknown-linux-gnu.tar.gz tar -C /opt/rv/${RV_VERSION}/ -xzvf /tmp/rv.tar.gz rm /tmp/rv.tar.gz chmod +x /opt/rv/${RV_VERSION}/rv ln -s /opt/rv/${RV_VERSION}/rv /usr/local/bin/rv /usr/local/bin/rv --version EOF RUN dnf clean all </code></pre> It should be pretty self-explanatory if you've seen a Containerfile before (or if you read Jim Gardner's post!), but I'll explain my choices a little bit. I originally explored using a Debian base image for my development environment, as I wondered whether Fedora's release cadence might be a little too fast for the purpose at hand, and Debian is usually the go-to for stability. However, after years of Fedora usage, I was pretty shocked at how out of date the software in the Debian repos (not just in stable</code>!) was compared to Fedora. So, in the end, I opted for Fedora 43 as a starting point, which should be perfectly fine for the foreseeable future, and requires less tinkering out of the box. The bulk of the Containerfile is basically just installing the software I want to be available out of the box. I use rig</code> for R installation management, and rig provides an RPM package for Fedora 43, making installation a breeze. Using rig, I make sure I have R 4.4.x and 4.5.x available in the container, the two versions I have projects with dependencies on. The rest is fairly straightforward if you know the shape of my workflow from previous posts. Neovim</a> for editing, lazygit</a> for lazy git, Quarto</a> for computational notebooks, uv</a> and rv</a> for managing Python and R projects. I chose to use the new kid on the block arf</a> as the R REPL, since it comes as a single binary that's a bit easier to install than radian</a>, which is a Python package (and is also no longer being actively maintained, unfortunately). After using arf for a few months, this was a good decision - it's great software. Normally, I'd install install a lot of these tools on my system by symlinking the binaries into ~/.local/bin</code>, but since we need to stay out of the home directory, I instead target /usr/local/bin</code> in the Containerfile. That's it, really. Not much else needs to be done. Toolbx or Distrobox handle all the configuration of entrypoints and mounts etc. Environment variables can be picked up from your dotfiles later. As long as I clone my Neovim config</a> into ~/.config/nvim</code> at some point (note that I could do this from the host), then when I run nvim</code> for the first time inside the toolbox, it will configure itself how I like and get all my plugins installed. If you really want to pre-configure some of the software, and the software allows config files to be placed in a system-level config file location, you could handle putting your config files there within the Containerfile. Just bear in mind that if you also have a config file in the user-level config file location, that will usually take precedence. I suppose there's also the question of pre-installing R packages somewhere outside of the home directory and making sure that library is added to your R profile. But I decided I'd rather just do a supervised installation of R packages after setup. Deployment</h3> You can build the image from the Containerfile by running: podman build --squash --tag localhost/f43custom . </code></pre> from the directory with the Containerfile in it. Then, if everything builds smoothly, you create your Toolbx with toolbox create --image f43custom workbox </code></pre> Or, in my case (the --unshare-process</code> flag is needed</a> on a systemd</code>-free system): distrobox create --image f43custom --name workbox --unshare-process </code></pre> and enter it with toolbox enter workbox</code> / distrobox enter workbox</code>. I also have an alias set up so that I can run distrobox enter workbox && nvim</code> in a brief command. Takeaways</h2> I should point out that the Containerfile above is still only an early attempt at this, and I'll probably continue to tinker around the edges of my setup, as usual. I am interested in whether I can get the size of the uncompressed image down, though I don't have the spare time to go down that rabbit hole just yet. For now, I'm pretty happy just to have something that works as intended. For my use case, the goal is not to constantly update the Containerfile and rebuild the container with the latest and greatest versions of the software. When a new version of arf or rv or Quarto is released and I want to move to it, I simply download it, enter the toolbox, and move the new version to /usr/local/bin</code> etc in the toolbox. No need to rebuild or anything. In general, when a new tool altogether is added to the container or a major change is made, then it's time to update the Containerfile to reflect that. It's about keeping the Containerfile in sync with the state of the container, so that I can feel ready to rebuild it quickly if I needed to. Obviously, this kind of thing is always going to end up being pretty specific to the needs and preferences of the individual user. But if it turns out that I'm not the only person that finds this approach useful, I wonder whether having a few curated starting images available would be a nice resource for the R/data science/computational social science community. The rocker</a> project has been a great resource for people, and I note that there is already some pretty good representation in the devcontainer</a> spec space for features from the data science world. Maybe a goal could be to build a stock of tried and tested Toolbx-compatible images that are tailored to data science workflows, and push these out to a container registry for others to use as a starting point. In the meantime, I've pushed mine to Docker Hub</a> in case anyone wants to try it out without having to build the image from the Containerfile. The repo</a> has the Fedora 43-based image, which is tagged f43, and it has an equivalent image which is Debian-based, tagged debian13. In any case, let me know if you found this post useful, or if you know of other tools that complement the kind of approach I've described here.

Using project scripts to keep Quarto source and output files organised

2024-10-14T14:00:00+00:00

In this post I just want to share a quick tip for how to solve a very specific problem when working with Quarto</a> projects: you want to keep your notebook source (.qmd) files and your notebook output (.pdf, .html etc) files separate, and you keep your source files in a subdirectory within the project. I'll talk a little bit about why that's a problem, and how to leverage one of Quarto's cool but lesser-known features to get around it.

The problem</h2>
Project-based workflows</h3>
If you're working with computational notebooks, you're probably also working with a project-based workflow. By this I really just mean you have a base directory for your project, you organise the files for your project in subdirectories, and you specify paths to files relative to the project base directory.
For an R user, the structure of a simple data analysis project called `my_project</code> might look something like this:`
my_project/ ├── data/ │ ├── raw/ │ │ └── raw_data.csv │ └── processed/ │ └── cleaned_data.csv ├── scripts/ │ └── clean_raw_data.R ├── figs/ │ └── fig1.png ├── .Rprofile └── README.md </code></pre> Generally, it's a good idea to keep different types of files used in the project in their own semantic subdirectories like this, rather than keeping them in the project base directory.
The same principle applies to projects that make use of Quarto notebooks too, and Quarto actually has built-in functionality for defining projects. Quarto projects</a> are marked by a `_quarto.yml</code> file in the project base directory, with project-level options and preferences being defined in this file.`
`However, if you keep your notebook source files in a subdirectory, similar to the scripts/</code> example above, Quarto's default behaviour will make it a little harder to keep the notebook files organised.`
So what's the issue with Quarto projects and subdirectories?</h3> When you render</code> a Quarto notebook, the rendered output file will by default be placed alongside the notebook source file. For small projects, this may not be a problem. However, as the number of notebooks and/or output formats starts to grow, the situation can quickly become quite messy. In these situations, it's helpful to have the rendered documents live in one folder, and the notebook source files that you edit in another folder.This is where the problem lies for Quarto projects. Quarto allows you to specify an output directory</a> in your project's metadata, in which case it will move output to that directory after rendering. However, it always considers any project subdirectories containing the notebook source file as part of the notebook's file name, so it will reproduce the subdirectory structure inside the output directory. Let's illustrate what I mean by this with an example. If you want your rendered notebooks to live in the quarto_output/</code> folder, you can specify output-dir: quarto_output</code> in the _quarto.yml</code> file. If you put your notebook my_notebook.qmd</code> in the project base directory and render</code> it to, say, a PDF document using Quarto, Quarto will place my_notebook.pdf</code> in quarto_output/</code>, which is great. However, if your source file is in scripts/my_notebook.qmd</code>, Quarto will place the rendered notebook in quarto_output/scripts/my_notebook.pdf</code>, which is less great, and probably isn't what most people meant when they specified output-dir: quarto_output</code>. It is, however, the intended behaviour</a>, and the Quarto devs have not indicated any intention to make the output-dir</code> option behave more like the user is likely to expect. How to solve the problem</h2> Hopefully, Quarto will eventually add a feature that provides a smoother way of keeping output and source files for notebooks separate in projects using a subdirectory for the notebook source. Until then, we can make use of Quarto's project script capabilities to take care of this. Project scripts</h3> As part of the rendering pipeline, Quarto allows users to specify pre- and post-render scripts</a>, which can integrate additional tasks into the process. There are lots of possible uses for these scripts, and one particularly helpful one for our purposes is automatically organising your project files after each render. To do this, all we need to do is create a script called, say, _move_quarto_output.R</code> that moves rendered notebooks from the source folder to a separate output folder. Then we just make sure that script is located in the project base directory and point post-render</code> to it in the project's _quarto.yml</code> metadata: project: post-render: _move_quarto_output.R </code></pre> The script will be called each time quarto render</code> is run, at the end of pandoc</a>'s rendering of the notebook to the specified output format. Note that pre- and post-render scripts in Quarto projects can be written in any language that Quarto can interpret, or can just be shell commands, giving you a tonne of flexibility. How my post-render script looks</h3> Here is the rather clunky script I use in my Quarto projects, which makes the following assumptions based on my own typical workflow (each of these assumptions can be relaxed by modifying the script): notebook source files are stored in the project subdirectory scripts_and_notebooks/</code>.</li> we want to move the rendered notebooks to a project subdirectory quarto_output/</code>, and we don't want to nest rendered notebooks in further subdirectories of quarto_output</code>.</li> all rendered notebooks are HTML files.</li> </ul> # Give quarto/pandoc time to ensure the creation of .html files is finished Sys.sleep(1.5) # Create `quarto_output` if it doesn't exist if (!dir.exists("quarto_output")) { dir.create("quarto_output") } # Grab the full file paths, allowing for a couple levels of directory nesting old_html_files <- c(Sys.glob("scripts_and_notebooks/.html"), Sys.glob("scripts_and_notebooks//.html"), Sys.glob("scripts_and_notebooks///.html")) if (length(old_html_files) > 0) { # Convert the old file paths to where we want them to go, # collapsing nested directories into the new file name new_html_files <- old_html_files |> stringr::str_replace_all("scripts_and_notebooks/", "quarto_output/") |> stringr::str_replace_all("(quarto_output/.)(/)(.)", "\\1_\\3") # Move the files (stored in a var as file.rename returns a logical vector) renamed <- file.rename(from = old_html_files, to = new_html_files) # Print the outcome to the screen if (!FALSE %in% renamed) { cli::cli_alert_success("{length(renamed)} file{?s} moved successfully:") for (i in 1:length(renamed)) { cli::cli_bullets(c( "", {old_html_files[i]}, ">" = cli::col_green({new_html_files[i]}) )) } } } else { cli::cli_alert_danger("Tried moving generated .html files but none were found.") } </code></pre> What the script does is grab the project-relative file paths of the rendered HTML files using base::Sys.glob()</code>, and then uses base::file.rename()</code> to modify these paths, effectively moving the files. Regular expressions are used (via stringr</a>) to incorporate any nested source subdirectories into the output file name (such that source/subdir/notebook1.qmd</code> results in output/subdir_notebook1.html</code>). The output from the script will be appended at the end of quarto render</code>'s CLI output, which is why I use functions from the cli</a> package to print a summary of the operation: That's it! Let me know @pete@fedi.petejon.es</a> if this helps or if there are even easier ways of dealing with this.

charinet: A package for character interaction network analysis

2021-09-24T10:18:00+00:00

As part of my ESRC postdoc fellowship I have been working on turning the tools I use in my research to analyse character interaction networks into an R package called charinet</a>.

I won't go into detail here about what the package does and how it works because the project README contains a pretty extensive demo. But the main things it offers are:

Functions for quickly reading in and reshaping lists of character interactions in common data structures.</li>

A screenplay parser which will extract a relational event list from a well-formatted PDF of a screenplay.</li>

An implementation of the dynamic narrative centrality measure proposed in our Social Networks paper</a>.</li>

Functions for visualising the character interaction data as heatmaps, networks and dynamic centrality trajectories.</li> </ul>
I've been using the package myself for the last few months and it has really improved my workflow for the common tasks that I usually want to do when analysing character interaction data.
It also works well in conjunction with the data already available in the movienetdata</a> package.
With just a few lines of code, you can get a good sense of the character dynamics of a given narrative:
iron_man <- movienetdata::mcu01_ironman # Plot as a network: plot_charinet(adjacency = iron_man$adjacency, char_names = iron_man$node_list$char_name, degree = iron_man$node_list$nlines, node_fill = ifelse(iron_man$node_list$char_female, "#ded649", "#55467a"), parallel_edges = TRUE, title = "Dialogue in Iron Man (2008)") # Plot as a character activity heatmap: iron_man$event_list %>% activity_heatmap(char_names = iron_man$node_list$char_name, story_chunk_col = 2, from = 3) %>% plot_heatmap(title = "Activity heatmap for Iron Man by scene") # Plot dynamic centrality scores: iron_man$event_list %>% narrative_centrality(char_names = iron_man$node_list$char_name, mode = "out", wp = 0.01, normalised = TRUE, from = 3) %>% plot_nc(title = "Speaking centrality scores for characters in Iron Man (2008)") </code></pre> I'd love for people to try it out and let me know if it works for them and if there's anything else they'd like a package like this to do. I've tried to document everything clearly in the package, but do let me know if any of the functions or arguments aren't clear.

info </div> Note </div> This post was edited to update the various references to the package source repository, which is now hosted on Codeberg</a>. </div> </div>

Implementing narrative centrality for character interaction networks: a practical walkthrough

2020-04-30T12:05:15+01:00

warning </div> Note from the future! </div> I later implemented the centrality measures discussed in this post in the {charinet}</a> package, and the functions there</a> provide a better and easier to use implementation which should supercede the code in this post.
Even though this post is outdated now, I'll leave it up because the discussion around the measures covers some slightly different ground to what's covered in the {charinet} README. </div> </div>
In this post I'll provide a practical walkthrough to complement a paper I wrote with my co-authors Johan Koskinen and Eithne Quinn on measuring centrality in character interaction networks, which has now been published in Social Networks (read the published version here</a> or the accepted author manuscript here</a>.).
I've been presenting this work at conferences for a while now, and have received some great feedback along the way. Now that the paper is finalised and published, it seemed like a good opportunity to share the code I use to implement the measure in R and show how to use it.
The paper in a nutshell</h2>
I won't try to recreate the paper within this post, as I'd rather stick to the practical side here and let readers learn more about the measure from the paper where it is properly explained and contextualised.
However, a very brief nutshell summary might help inform what follows. Broadly speaking, the paper reflects on the relationship between network centrality and narrative centrality. In particular, it argues that in order for a network analytic measure to tell us about a narrative text, it needs to be able to reflect the sequence of events that take place in the narrative. In other words, we need to represent the narrative as a dynamic network, and our measures ought to be designed to work on such a dynamic representation.
To this end, we propose a simple dynamic measure which allows characters' scores to increase when they either send or receive a relational event (in our case, when they either speak or are spoken to) in a way that reflects the characters with whom they interact as well as the sequence of such interactions.
This measure, which has an "out" version and an "in" version to reflect the directionality of interaction data, looks like so:

We argue that this approach reveals much more about the dynamics of a narrative text because it allows us to say something meaningful about the interlocking trajectories which characters take through narratives - trajectories which are often extremely difficult to summarise using a single static value.
What does applying the measure look like in practice?</h2>
The easiest way to explain how to use the measure is to work through a practical example. So, following the paper, I'll use the measure to calculate the narrative centrality of characters in Star Wars: The Force Awakens (2015) using character interaction data I collected in my PhD research. The Force Awakens is one of 30 films available in the {movienetdata}</a> package, so it will make for an easy demonstration.
Defining the function</h4>
First, we need to define a function for calculating the centralities. The goal is to calculate a score for each character at each time point, so essentially we want an N-by-T matrix, where N is the total number of named speaking characters and T is the total number of narrative events (in my case, dialogue interactions). In fact, we want two such matrices - one for the speaking scores and one for the spoken-to scores.
I define the function in a way that allows the user to specify which matrix they want via the mode</code> argument, which can take the values "in" (for the spoken-to scores), "out" (for the speaking scores) or "both". In the case of "both", which is the default option, a named list is returned containing first the "out" matrix and then the "in" matrix. For either "out" or "in", a single matrix is returned.
The weighting parameter lambda can be specified via the argument wp</code>, which has a "sensible default" of 0.01 (see the Appendix of the paper for a detailed discussion of this).
The argument normalised</code> allows the user to specify whether to return the normalised scores (normalised = TRUE</code>) or the absolute scores (normalised = FALSE</code>). Normalising the scores here refers to dividing each character's score at time t by the sum of all character scores at time t. The default value is TRUE</code>.
Finally, I want the function to be flexible with regard to the structure of the event list on which it is applied. To provide a balance between this flexibility and creating some predictability regarding the input data structure, the function assumes an event list (given via the event_list</code> argument) containing a column of speaker IDs in the first column and dummy variables for each potential receiver in column recipient ID + 1</code>. So Character 3 will be represented by column 4, such that a value of 1 in cell [3, 4]</code> means that Character 3 was the recipient of the line of dialogue spoken at time t=3</code>. In some cases, the input data will contain other columns. For example, the {movienetData} event lists also contain columns eventID</code> and sceneID</code>. The argument start_at</code> allows the user to skip any such columns so long as they are positioned prior to the speaker ID. Basically, the value given for this argument should be the speaker ID column. So if the speaker ID column is first in the input event list, start_at</code> should be set at 1 (which is the default), but if the speaker ID column is column 3 in the input event list (as, for example, in the {movienetData} data), start_at</code> should be set at 3. This will become clearer in the practical examples below.
This might seem at first like an unusual assumed structure, but I prefer this dummy-based event list format to a (perhaps more conventional) senderID-recipientID edgelist-style format because the data is characterised by sender exclusivity and receiver non-exclusivity. That is, there is only ever one sender per event, but they might be addressing multiple recipients simultaneously (as is the case, for example, with e-mail or tweet data). The event list allows this multi-recipient structure to be reflected clearly in the data - one row per event - and streamlines both the data collection process and the actual computation when using the measure. Any additional variables for each event (for example, weight or type may be relevant in other contexts) can easily be included before the speaker ID column, and start_at</code> ensures that differences in the number or name of such variables won't make analysis any more complicated.
An optional character vector containing the names of the characters corresponding to the node IDs can be passed in via the chars</code> argument, and these names will be used to supply column names in the returned matrices. If these are not provided, character names will be inferred from the input event list.
All together, this looks like so (I've also made this available as a gist</a>):
narrative_centrality <- function(event_list, chars = NULL, mode = "both", wp = 0.01, normalised = TRUE, start_at = 1) { C_in <- matrix(1, nrow(event_list), length(chars)) C_out <- matrix(1, nrow(event_list), length(chars)) C_in_norm <- matrix(1, nrow(event_list), length(chars)) C_out_norm <- matrix(1, nrow(event_list), length(chars)) if(is.null(chars)) { chars <- colnames(event_list)[start_at + 1:ncol(event_list)] } colnames(C_in) <- chars colnames(C_in_norm) <- chars colnames(C_out) <- chars colnames(C_out_norm) <- chars # Compute the scores C_in_t <- matrix(1, length(chars), 1) C_out_t <- matrix(1, length(chars), 1) for (t in 1:nrow(event_list)) { speaker <- event_list[t, start_at] receivers <- which(event_list[t, (start_at + 1):dim(event_list)[2]] == 1) C_in_t[receivers] <- C_in_t[receivers] + (wp * C_in_t[speaker]) C_out_t[speaker] <- C_out_t[speaker] + ((wp / length(receivers)) * sum(C_out_t[receivers])) for (c in 1:length(chars)) { C_in[t, c] <- C_in_t[c] C_out[t, c] <- C_out_t[c] C_in_norm[t, c] <- C_in_t[c] / sum(C_in_t) C_out_norm[t, c] <- C_out_t[c] / sum(C_out_t) } } # Return the requested values according to `normalised` and `mode` if(normalised == TRUE) { if(mode == "both") { return(list("out_scores" = C_out_norm, "in_scores" = C_in_norm)) } if(mode == "in") { return(C_in_norm) } if(mode == "out") { return(C_out_norm) } } else { if(mode == "both") { return(list(C_out, C_in)) } if(mode == "in") { return(C_in) } if(mode == "out") { return(C_out) } } } </code></pre> So to get the centralities for The Force Awakens, we need to read in the film data: # remotes::install_git("https://codeberg.org/pjphd/movienetdata.git") tfa <- movienetdata::starwars_tfa </code></pre> and, just to be clear, this event list is structured like so: tfa$event_list[1:5, 1:7] </code></pre> ## eventID sceneID speakerID Lor.San.Tekka..1. Poe.Dameron..2. Kylo.Ren..3. Captain.Phasma..4. ## [1,] 1 1 1 0 1 0 0 ## [2,] 2 1 2 1 0 0 0 ## [3,] 3 1 1 0 1 0 0 ## [4,] 4 1 2 1 0 0 0 ## [5,] 5 1 2 1 0 0 0 </code></pre> our speaker ID column is column 3, and our dummy variables for each recipient start at column 4 and occupy the remaining columns in the dataset (here I only show the first few for readability). So now we can run the function, remembering to start at column 3: tfa_scores <- narrative_centrality(tfa$event_list, chars = tfa[[2]]$char_name, wp = 0.01, start_at = 3) </code></pre> On its own, the returned matrix is a little too unwieldy to offer immediate insights into what's going on. So we can plot the scores to see the character trajectories more clearly. First, I'll put the data into a tidy</a> format to make things easier. In the process, I'll add a variable which makes sure that only those characters whose share of scores at the end of the film is greater than their share at the start of the film are labelled, which will make the graph a little more readable. library(tidyverse) </code></pre> tfa_out_tidy <- cbind.data.frame("Event" = seq.int(1, dim(tfa_scores$out_scores)[1]), tfa_scores$out_scores) %>% tidyr::pivot_longer(cols = -Event, names_to = "Character", values_to = "Score") %>% mutate(Label = ifelse(Character %in% names(which(tfa_scores$out_scores[1, ] < tfa_scores$out_scores[dim(tfa_scores$out_scores)[1], ])), Character, NA)) tfa_in_tidy <- cbind.data.frame("Event" = seq.int(1, dim(tfa_scores$in_scores)[1]), tfa_scores$in_scores) %>% tidyr::pivot_longer(cols = -Event, names_to = "Character", values_to = "Score") %>% mutate(Label = ifelse(Character %in% names(which(tfa_scores$in_scores[1, ] < tfa_scores$in_scores[dim(tfa_scores$in_scores)[1], ])), Character, NA)) </code></pre> So the data now looks like so: tfa_out_tidy </code></pre> ## # A tibble: 21,328 x 4 ## Event Character Score Label ## <int> <chr> <dbl> <chr> ## 1 1 Lor San Tekka 0.0326 <NA> ## 2 1 Poe Dameron 0.0322 Poe Dameron ## 3 1 Kylo Ren 0.0322 Kylo Ren ## 4 1 Captain Phasma 0.0322 <NA> ## 5 1 Finn 0.0322 Finn ## 6 1 Unkar 0.0322 <NA> ## 7 1 Rey 0.0322 Rey ## 8 1 General Hux 0.0322 <NA> ## 9 1 Lieutenant Mitaka 0.0322 <NA> ## 10 1 Han Solo 0.0322 Han Solo ## # ... with 21,318 more rows </code></pre> And then this can be plotted. First, the speaking scores: library(directlabels) </code></pre> tfa_out_tidy %>% ggplot(aes(x = Event, y = Score, group = Character, colour = Label)) + geom_line(size = 1, show.legend = FALSE) + scale_x_continuous(expand = expand_scale(c(0, 0.13))) + geom_dl(aes(label = Label), method = list("last.bumpup", fontface = "bold")) + theme_light() + theme(axis.line = element_line(colour = "black"), panel.grid.major = element_blank(), panel.grid.minor = element_blank(), panel.border = element_blank(), panel.background = element_blank()) </code></pre> and then the spoken-to scores: tfa_in_tidy %>% ggplot(aes(x = Event, y = Score, group = Character, colour = Label)) + geom_line(size = 1, show.legend = FALSE) + scale_x_continuous(expand = expand_scale(c(0, 0.13))) + geom_dl(aes(label = Label), method = list("last.bumpup", fontface = "bold")) + theme_light() + theme(axis.line = element_line(colour = "black"), panel.grid.major = element_blank(), panel.grid.minor = element_blank(), panel.border = element_blank(), panel.background = element_blank()) </code></pre> Here we can see the same graphs used in the paper. They illustrate many of the strengths of the measure. For example, we can see that some characters' importance is very much temporally situated - as with Poe Dameron, who is pivotal in initiating the action, but disappears after the first act to return later in the film after other characters have established themselves as the primary narrative actors. This illustrates a second desirable property of the measure: when a character stops participating (in this case vocally), their share of the "character space" declines, as we see with characters like Poe and Maz Kanata, who come and go, but also with Rey when she is kidnapped and becomes temporarily much more passive in the second act of the film. To illustrate the effect of different values of the weighting parameter lambda (again, see the paper for full discussion of this), let's recalculate and re-plot the scores with a higher lambda value of 0.05. tfa_scores_2 <- narrative_centrality(tfa$event_list, chars = tfa[[2]]$char_name, mode = "out", wp = 0.05, start_at = 3) cbind.data.frame("Event" = seq.int(1, dim(tfa_scores_2)[1]), tfa_scores_2) %>% tidyr::pivot_longer(cols = -Event, names_to = "Character", values_to = "Score") %>% mutate(Label = ifelse(Character %in% names(which(tfa_scores_2[1, ] < tfa_scores_2[dim(tfa_scores_2)[1], ])), Character, NA)) %>% ggplot(aes(x = Event, y = Score, group = Character, colour = Label)) + geom_line(size = 1, show.legend = FALSE) + scale_x_continuous(expand = expand_scale(c(0, 0.13))) + geom_dl(aes(label = Label), method = list("last.bumpup", fontface = "bold")) + theme_light() + theme(axis.line = element_line(colour = "black"), panel.grid.major = element_blank(), panel.grid.minor = element_blank(), panel.border = element_blank(), panel.background = element_blank()) </code></pre> Here, the number of interactions required for one character to catch up to another character is vastly reduced, rendering the measure much more sensitive to the effect of each interaction as reflected in the spiky lines. This means that characters both climb and decline more quickly, allowing characters to "steal the show" by becoming active late in the film (particularly if they interact with hitherto central characters), and be forgotten quickly by leaving the action early. Personally, having experimented with running the measure on many different films, I find that the trajectories given by the measure tend to align much more closely with my mental model of a narrative when the value is lower than in the above example, as I think my understanding of the narrative is much less volatile than the spiky picture painted by scores calculated with higher lambda values. Some exploratory work will be needed to find appropriate values in different contexts and cases. Computational extraction and data caught in the wild</h2> I make the point in the paper that the principles of the measure are independent from the particular data extraction method we use to illustrate them. In other words, all the arguments are just as applicable to computationally extracted character networks, which are the most commonly pursued in the data science and digital humanities communities. What's important is not whether data are extracted manually or computationally, but rather that extracted data are stored in an event list format rather than going straight into a static edgelist or adjacency matrix. The edgelist and adjacency matrix can be derived from the event list later, but the event list can never be reverse-engineered from the static representations. To illustrate this, I'll provide a final example of the measure at work using data from the {schrute}</a> package which contains dialogue from episodes of the US version of The Office. # install.packages("schrute") office_data <- schrute::theoffice </code></pre> Which is formatted like so: office_data %>% dplyr::filter(season == 1) %>% dplyr::filter(episode == 1) %>% dplyr::slice(1:3) </code></pre> ## # A tibble: 3 x 12 ## index season episode episode_name director writer character text ## <int> <int> <int> <chr> <chr> <chr> <chr> <chr> ## 1 1 1 1 Pilot Ken Kwa~ Ricky~ Michael All ~ ## 2 2 1 1 Pilot Ken Kwa~ Ricky~ Jim Oh, ~ ## 3 3 1 1 Pilot Ken Kwa~ Ricky~ Michael So y~ ## # ... with 4 more variables: text_w_direction <chr>, imdb_rating <dbl>, ## # total_votes <int>, air_date <fct> </code></pre> We can create a character network for the first episode by first subsetting the data: ep_1 <- office_data %>% dplyr::filter(season == 1 & episode == 1) %>% dplyr::select(index, character) %>% # This next line just fixes a typo I found in the data for a character name dplyr::mutate(character = str_replace(character, "Michel", "Michael")) </code></pre> Now we need to transform this into an event list. As we only have the speaker and not the direct recipient of the data, how should we determine which characters are the js to use when increasing speaker i's score? We can approximate receipt of dialogue by assigning any other characters who spoke within some number of lines (here, I'll use 3) either side of the speaker as a "receiver" of the dialogue. This transformation looks like so: to_events <- function(data) { num_chars <- length(unique(data$character)) get_id <- function(char_name = "") { which(unique(data$character) == char_name) } events <- matrix(0, nrow = nrow(data), ncol = 2 + num_chars) colnames(events) <- c("eventID", "speakerID", unique(data$character)) for (x in 1:nrow(data)) { events[x, 1] <- x events[x, 2] <- get_id(data$character[x]) for(c in unique(data$character)) { if(c %in% data$character[max(x - 3, 1):min(x + 3, nrow(data))]) { events[x, 2 + get_id(c)] <- 1 events[x, 2 + events[x, 2]] <- 0 # No self-ties! } } } return(events) } </code></pre> Which produces a familiar data structure: ep_1_events <- to_events(ep_1) ep_1_events[1:5, 1:7] </code></pre> ## eventID speakerID Michael Jim Pam Dwight Jan ## [1,] 1 1 0 1 0 0 0 ## [2,] 2 2 1 0 0 0 0 ## [3,] 3 1 0 1 0 0 0 ## [4,] 4 2 1 0 0 0 0 ## [5,] 5 1 0 1 1 0 0 </code></pre> Now we can run the centrality measure on this (we only need the "out" scores because there is no true recipiency in this data): office_scores <- narrative_centrality(ep_1_events, chars = unique(ep_1$character), mode = "out", start_at = 2) cbind.data.frame("Event" = seq.int(1, dim(office_scores)[1]), office_scores) %>% tidyr::pivot_longer(cols = -Event, names_to = "Character", values_to = "Score") %>% mutate(Label = ifelse(Character %in% names(which(office_scores[1, ] < office_scores[dim(office_scores)[1], ])), Character, NA)) %>% ggplot(aes(x = Event, y = Score, group = Character, colour = Label)) + geom_line(size = 1, show.legend = FALSE) + scale_x_continuous(expand = expand_scale(c(0, 0.13))) + geom_dl(aes(label = Label), method = list("last.bumpup", fontface = "bold")) + theme_light() + theme(axis.line = element_line(colour = "black"), panel.grid.major = element_blank(), panel.grid.minor = element_blank(), panel.border = element_blank(), panel.background = element_blank()) </code></pre> And, to illustrate my point about deriving the static representations later, here is the sociogram for the episode after aggregating into an adjacency matrix: library(igraph) library(ggraph) library(graphlayouts) </code></pre> office_adj <- matrix(0, nrow = length(unique(ep_1$character)), ncol = length(unique(ep_1$character))) for (i in 1:length(unique(ep_1$character))) { for (j in 1:length(unique(ep_1$character))) { office_adj[i, j] <- length(which(ep_1_events[, 2] == i & ep_1_events[, j + 2] == 1)) } } g <- graph_from_adjacency_matrix(office_adj, weighted = TRUE, diag = FALSE) for (x in 1:length(unique(ep_1$character))) { V(g)$nlines[x] <- length(which(ep_1_events[, 2] == x)) } V(g)$name <- unique(ep_1$character) V(g)$size <- scales::rescale(V(g)$nlines, to=c(1.5,14)) g %>% ggraph(layout = "stress") + geom_edge_link(aes(alpha = weight, start_cap = circle(node1.size * 1.1, unit = "pt"), end_cap = circle(node2.size * 1.1, unit = "pt")), n = 2, arrow = arrow(angle = 20, length = unit(0.1, "inches"), ends = "last", type = "closed"), colour = "#e86682") + geom_node_point(size = V(g)$size, shape = 21, stroke = 0.5, fill = "#eb4034", colour = "black") + geom_node_text(label = V(g)$name, size = 4, repel = TRUE) + scale_edge_alpha_continuous(range = c(0.2, 1)) + labs(title = paste0("Dialogue ties in The Office S01E01")) + theme_graph() + theme(legend.position = "none", plot.title = element_text(size = 16, hjust = 0.5)) </code></pre> Wrapping up</h2> Hopefully this walkthrough helps make the measure less abstract and more accessible/user-friendly, but if I can clarify anything further, do get in touch and I'll do my best to help. As a final note, the measure is intended as a simple starting point on which further dynamic measures can be built. Some assumptions can be relaxed if this makes sense in other applications and contexts (for example, network actors don't have to start with identical scores if we have prior information). Additional terms can be added where this makes sense (for example, allowing characters' scores to increase when they are not directly involved in an interaction but are the subject of an interaction, such as when A talks to B about C). I will be exploring some of these different options myself, and I hope others will too.

movienetdata: A data package for character interactions in popular films

2020-03-03T15:28:00+00:00

I recently released the data I collected during my PhD research as a data package</a> for R. The main reasons I did this are:

Sharing

One of the most common questions I get when chatting to people at conferences etc. is whether my data is publicly available. I assume this means that people would be interested in taking a look at the data, analysing it, playing around with it and such.

The dataset was generated to answer questions</a> about the narrative marginalisation of women in popular cinema. However, there are a number of reasons why somebody might want to use it beyond this purpose:

It is an ideal dataset for exploring character networks in all their complexity - each film in the dataset comes with both an event list, which can be used for constructing dynamic network representations of the narratives, and an adjacency matrix derived from this event list, which can be used for static network representations. The event list also indexes each interaction by scene, allowing for the interaction data to be grouped into story chunks.</li>
The dataset offers useful teaching data for illustrating basic network analysis ideas using familiar, easy to grasp examples of networks.</li>
I'm sure there are all kinds of interesting questions about the films that the data can help to answer that I didn't think to ask.</li> </ul>
Making my life easier
I plan on sharing other aspects of my character network analysis work, and releasing the data this way allows me to separate out the data from the core functionality of the various tools and functions that I work with. This is useful as it helps me to ensure that they make sense independently of the particular data with which I have been using them to work.
Finally, I wanted some hands-on experience with R package development, and data packages offer a great way to learn the basics of package structure and documentation.
How to use the data</h2>
In R
If you use R, you can install the package using any package manager that can install from git repositories:
# e.g. using {remotes}: remotes::install_git("https://codeberg.org/pjphd/movienetdata.git") # or using {pak}: pak::pkg_install("git::https://codeberg.org/pjphd/movienetdata.git") </code></pre> Then, load the package:
library(movienetdata) </code></pre> You can browse the complete list of different films in the dataset using data(package = "movienetdata")</code>: And load a specific film into the workspace using e.g. data("mcu01_ironman", "movienetdata")</code>. Each film is stored as a list containing three elements:
The event list. Each row contains information on a line of dialogue, with columns corresponding to eventID</code>, sceneID</code>, speakerID</code>, and dummy variables for each recipient.</li> The node list. Rows correspond to named speaking characters, with columns corresponding to variables for ID, name, total lines spoken (nlines</code>), total times spoken to (linesin</code>), and gender. Note that some films in the dataset might have additional or different variables included.</li>
The adjacency matrix. This is derived from the event list through aggregation and can be useful for static purposes such as visualising the network as a sociogram.</li> </ol> Access specific elements by name or by indexing the list:# Get the nodes by name mcu01_ironman$node_list </code></pre> # Get the nodes by indexing mcu01_ironman[[2]] </code></pre> Example: If you just want to create a quick network diagram for the film Black Panther, you can do something like this: black_panther <- movienetdata::mcu18_blackpanther # install.packages("network") library(network) # Create network object from adjacency matrix bp_net <- network(black_panther$adjacency) # Plot the network plot(bp_net, label = black_panther[[2]]$char_name, edge.col = "lightpink1", vertex.cex = sqrt(black_panther[[2]]$nlines)/3, label.pos = 3, arrowhead.cex = 0.7, vertex.col = ifelse(black_panther[[2]]$char_female == 1, "#ded649", "#55467a"), main = "Character interactions in Black Panther (2018)") </code></pre> Not in R If you're not an R user, you can still access the raw data from the package source repository</a>. In the data-raw</code> folder you will find the CSV files for each film's event list and node list. info </div> Note </div> This post was edited to update the various references to the package source repository, which is now hosted on Codeberg</a>. </div> </div>

Step 1: find a guinea pig</h4>
Or, more specifically, a bear. I needed an image to experiment on, so I went to Unsplash</a> and found this magnificient bear</a> to participate in my trials:</p>
</p>

Step 6: celebrate (or not...)</h4>
Actually, running the `glitch_it()</code> function a few times reveals a problem with this method: the resulting images often corrupt in a way which destroys whole sections of the image with grey blocks:</p>`
`</p>`

Pete Jones

You don't need CI to deploy a static website

Using toolbox containers as a persistent, stable and easy-to-deploy development environment for data analysis work

A brief guide to using Neovim as an R (and Quarto) development environment

Using project scripts to keep Quarto source and output files organised

charinet: A package for character interaction network analysis

Implementing narrative centrality for character interaction networks: a practical walkthrough

Making glitch art in R

movienetdata: A data package for character interactions in popular films

Pete Jones

You don't need CI to deploy a static website

Using toolbox containers as a persistent, stable and easy-to-deploy development environment for data analysis work

The pros and cons of a semi-containerised life</h3> Regardless of whether any of the above scenarios apply to you, I think I can boil down the approach to the following lists of pros and cons.</p> Some advantages of the toolbox model:</p>

A brief guide to using Neovim as an R (and Quarto) development environment

Static code analysis: LSP and Tree-sitter</h3> Many of the most useful features that editors offer are based on parsing and understanding your code in order to highlight it, lint it, keep track of document symbols, suggest completions and let you know if something doesn't look right.</p>

Using project scripts to keep Quarto source and output files organised

charinet: A package for character interaction network analysis

Implementing narrative centrality for character interaction networks: a practical walkthrough

Making glitch art in R

Defining a glitch method</h2>

Step 5: look at the results</h4> Now we want to take a look at what we've just created without leaving R.</p> We can use magick to do this, as magick can read raw data, and will print this image to the RStudio Viewer pane:</p>

Step 6: celebrate (or not...)</h4> Actually, running the glitch_it()</code> function a few times reveals a problem with this method: the resulting images often corrupt in a way which destroys whole sections of the image with grey blocks:</p> </p>

movienetdata: A data package for character interactions in popular films

Step 6: celebrate (or not...)</h4>
Actually, running the `glitch_it()</code> function a few times reveals a problem with this method: the resulting images often corrupt in a way which destroys whole sections of the image with grey blocks:</p>`
`</p>`