The Dolt Workbench is an open-source SQL workbench supporting MySQL, Postgres, Dolt, and Doltgres databases. The workbench is under active development, which means we have a fairly regular release cadence. Each new workbench release comes with separate builds targeting each of the following platforms:

• Mac
• Mac App Store
• Windows
• Windows App Store
• Linux
• Docker

Previously, we handled the packaging and build steps for each of these six platforms entirely manually. If you're curious, you can read more about that in this blog, but the gist is that it was a time-consuming process that resulted in a lot of lost developer time. In this article, we'll go over how we used GitHub Actions to build a pipeline that automated it down to a single button press.

Goals

Going into this, we wanted to accomplish a few different goals:

1. Speed - This is an obvious one, but the main disadvantage of the former build process was that it was slow and manual. Additionally, the actual build time for each of our target platforms ranged anywhere from 10 to 30 minutes. We wanted to parallelize this as much as possible while also taking advantage of cache support provided by GitHub Actions.
2. Reproducibility: Before building this pipeline, we ran all the build, packaging, and codesigning processes on our local machines. The build process was tightly coupled to the host machine with no centralized secret repository or environment variable manager. We wanted to create a highly reproducible build environment.
3. Retryability: One of the benefits of a manual build process is that if one step fails, you can very easily retry only that step without repeating the entire workflow. In a fully automated CD process, you are often forced to sacrifice this convenience in favor of speed. That said, a goal of ours was to ensure that retryability was at least as robust as parallel execution. For instance, if one of our requests to the Apple Store API times out, the Mac builds should fail, but all the others should continue on without issue. Afterwards, the Mac builds can be retried in isolation and incorporated into the same release.

With these goals in mind, let's get into how it actually works.

The Pipeline

This is the handy pipeline graph that GitHub generates for you, split into four separate sections.

Let's dive a little deeper into each of these steps.

1. Version

The Dolt Workbench follows the popular Semantic Versioning specification, which uses the MAJOR.MINOR.PATCH format. The sole input to our release workflow is the version number you would like to use for the new release. This version number is written to our package.json file before the rest of the build steps begin. Additionally, each of our target platforms has an associated build version (separate from the package version). The combination of package version and build version must be monotonically increasing between releases in order for the Apple and Microsoft stores to accept it.

Since build versions are in no way user-facing, we handle this restriction by automatically doing a PATCH bump on each new release for each platform build version. These version bumps are handled by a custom composite action that uses the helpful action-bump-semver utility and optionally commits the change. This snippet shows the main version bump logic:

- name: Get current build version
  id: current-build-version
  run: |
      version="$(yq '.buildVersion' ${{ steps.filepath.outputs.build-config-file }})"
      echo "Current buildVersion in ${{ steps.filepath.outputs.build-config-file }} is $version"
      echo "version=$version" >> $GITHUB_OUTPUT
  shell: bash

- name: Bump version
  uses: "actions-ecosystem/action-bump-semver@v1"
  id: bump-version
  with:
    current_version: ${{ steps.current-build-version.outputs.version }}
  level: patch

- name: Update ${{ steps.filepath.outputs.build-config-file }} buildVersion
  run: |
    yq -i '.buildVersion = "${{ steps.bump-version.outputs.new_version }}"' ${{ steps.filepath.outputs.build-config-file }}
    echo "New buildVersion in ${{ steps.filepath.outputs.build-config-file }} is ${{ steps.bump-version.outputs.new_version }}"
  shell: bash

Feel free to use/modify our bump-build-version action for your own applications!

2. Release

This step is relatively straightforward. We use the version number to create a git tag from the current main branch, then mark that tag as a release in GitHub. Check out our fork of the official create-release action for more details.

3. Build

This is the most important step in our outer release workflow. It kicks off six separate jobs that run concurrently. The first of these is our custom release notes generator, which you can read more about in this blog. The next is a separate workflow we use to build and push the Docker image for the new release to Docker Hub. Finally, we have four callable workflows for Mac, Mac App Store, Windows, and Linux that handle the full build process for their respective platforms. Each of these utilizes a GitHub runner corresponding to the target build platform (i.e. the Linux build scripts run on an Ubuntu runner).

Although a few steps are shared, we chose to separate these into different workflows in order to reduce complexity and allow for manual workflow dispatch at the platform level. Let's take a closer look at the steps involved for the Linux workflow, which is a good template for how the rest of them work.

A fair amount of this is boilerplate environment setup — installing Node, running our installation scripts, downloading Dolt, etc. We wrote another composite action shared across all four workflows that handles dependency installation. The most intensive step in the process is the actual build script, which reads the appropriate build configuration file and packages the app for distribution. The last step uploads the final executable as an asset to the GitHub release created in the previous step. If all goes well, the release should look something like this:

Mac and Windows are a bit more stringent with what they allow you to download on their machines, and as a result your application must be properly code-signed before it can be distributed on their platforms. For Windows, this is handled entirely by our build script. For Mac, however, the certificates and provisioning profiles used for code-signing must be issued by Apple for your app exclusively. This means that before the Mac build scripts can run, you must first download the correct provisioning profile and import the appropriate certificates into the MacOS runner's keychain.

For the provisioning profiles, we're using this action, which wraps the App Store Connect API and downloads the profiles from your developer account onto the runner. For the certificates, we're storing the base64-encoded versions of them as GitHub Actions secrets, and then loading those straight into the runner's keychain using the import-codesign-certs action. Once this is done, the build scripts will automatically locate the appropriate certificates and code-sign the app correctly.

The final executables produced for Mac and Windows app stores are not actually runnable via direct download. Instead, they're built specifically for distribution through the app stores. Because of this, those files are uploaded directly to the workflow run as GitHub artifacts, and only the direct-download packages (.dmg for Mac and .exe for Windows) are attached to the release.

4. Merge

The final step in the process is merging all the version bumps back into the main branch. This is done after all the other jobs complete so that (1) a single PR can capture all the changes, and (2) we don't accidentally commit a version bump for a build that eventually fails. If everything succeeds, a new PR will be created and merged with five commits — one for the package.json update, and one each for the Linux, Windows, Mac, and Mac App Store build versions.

Note that "[skip ci]" in the commit message causes GitHub to ignore our automated testing workflows that typically run when PRs are raised. If one of the builds fails, you can just rerun the workflow for the failing build and it will attach the appropriate files to the existing release and create/merge a separate PR containing only the individual version bump.

Future Work

Today, the only part of the release process left to automate is the actual upload to the official Mac and Windows stores. Both Apple and Microsoft expose APIs that allow third-party developers to create store submissions, and we've done some initial work on leveraging those for our own release process. We've also investigated Fastlane, another GitHub-Actions-style workflow runner that appears to be especially geared towards the Apple ecosystem. Stay tuned for an update on how we level up our release process using these tools!

To see the full Github Actions workflow implementations, check out the source code here. If you have any questions about our release process, or Dolt in general, we'd love for you to drop by our Discord.