Dolt Interactive Rebase
Dolt is the world's first fully-versioned, relational database. The inspiration for Dolt's decentralized versioning model is Git. Dolt allows you to branch, merge, diff, push, and pull your relational data in all the same ways that Git allows you to work with source code files. Today, we're excited to announce a major Git feature that is now available in Dolt – rebase!
Rebase is a powerful concept in Git and a feature we've wanted to bring to Dolt for a while. We're excited to have it available now, and we think you're going to enjoy using it! In this post, we'll start off by explaining how rebasing works, in both Git and Dolt, then we'll show an example of a Dolt interactive rebase where we clean up a messy commit history before we open a pull request with our changes on DoltHub.
What is Rebasing?
Let's start off with some basics on what rebasing is and how it works. Most Git users know about rebase and that it can rewrite history, but it's not as commonly used as merge or many other Git commands. Rebase is quite a powerful tool, and as with all powerful tools, it's important to know how it works so you can use it safely and effectively.
Commits and Branches
Let's start by looking at some rebase syntax and explain what it means. We'll be showing Dolt's syntax to rebase support, but the concepts are exactly the same for rebasing in Git.
call dolt_rebase('--interactive', 'main');In the example above, we're using the new dolt_rebase() stored procedure to start a rebase. The first argument, --interactive (alternatively, you can just use -i), tells Dolt that we want this to be an interactive rebase where we get a chance to examine and adjust the rebase plan before our commits are rebased. The second argument, main, is the upstream branch that we want to rebase onto. The currently checked out branch is implicitly the branch being rebased.
When we run this statement, Dolt will create a default rebase plan in the dolt_rebase system table that contains all the commits that are reachable from our current branch (i.e. the branch being rebased) but are NOT reachable from the upstream branch (i.e. the branch we're rebasing onto). This is the same as the "two-dot syntax" for Git and Dolt log.
In Git and Dolt, the default rebase plan is to replay all the commits unique to the branch being rebased, but to apply them to the tip of the branch we're rebasing onto (i.e. main in this example). With Git, you edit the rebase plan in a text representation, and with Dolt, you edit the rebase plan in the dolt_rebase system table. We'll see a more involved example later in this post that shows how to adjust the rebase plan to change the commit history.
One difference in Dolt's rebase support is that when you start a rebase, your session is moved to a new branch where the rebase work is applied. Dolt uses this "rebase working branch" instead of working in detached head mode, like Git does. In this example, since we're rebasing the branch1 branch, Dolt will temporarily switch over to a dolt_rebase_branch1 branch to let you modify the rebase plan and then it will be applied on this branch. When the rebase is successful, the branch being rebased will be updated to point to the final commit in the rebase chain. If you need to reconnect and continue a rebase operation, you can simply checkout this rebase working branch.
When we're ready to execute the rebase plan, we run call dolt_rebase('--continue');, and when the rebase finishes, our commit history looks slightly different. A chain of new commits is created off of the commit at the tip of the main branch, and our current branch ref is updated to point to the final commit in that chain. Note that the old commits still exist in our commit graph, but they are no longer reachable from any branch and will be removed the next time garbage collection runs, unless a branch or tag is set to point to them.
Cherry-Picking
At the heart of executing a rebase plan, is the ability to cherry-pick a commit. Cherry-picking a commit means looking at the changes in a commit, and then applying that exact same diff on top of another commit, in order to create a new commit. One important point to note is that this new commit will have a different commit ID from the original commit you cherry-picked, even though they have the same diff. This is because the commit ID is a hash of not just the rows in the diff but the entire database at the new commit, and also includes the commit message, author, date, and parent commit IDs.
Thanks to Jennifer, Dolt has supported cherry-picking commits for a while now, so building Dolt's rebase support on top of the existing cherry-pick functionality was pretty straightforward.
A good mental model for understanding how rebase works is to think of rebase as an engine for running a sequence of cherry-picks. The rebase plan lists which commits to cherry-pick, including in which order to cherry-pick them, which commits to skip, and how to alter commit messages for those cherry-picked commits.
Rebasing with Dolt
Now that you know the basics of how rebasing works in both Git and Dolt, let's walk through a more detailed example using Dolt's interactive rebase support. In the example below, we have a feature branch that we've been working on, and we want to use an interactive rebase to clean up our commit history before we open up a pull request to get our changes merged back into the database's main branch on DoltHub.
Our commit history is a bit of a mess! We've been working on our own development branch for some updates to our employee database and we want to get our commit history cleaned up before we open a pull request for our team to review and merge in the changes.
Let's take a look at our current commit history:
employees> select * from dolt_log;
+----------------------------------+-----------+-----------------+---------------------+---------------------------------------------------------------------+
| commit_hash | committer | email | date | message |
+----------------------------------+-----------+-----------------+---------------------+---------------------------------------------------------------------+
| 1kabuobsb2ouhlcb7se0gitcij90kll5 | root | root@localhost | 2023-12-28 19:32:08 | fixing typos in Franklin Fameemo's name |
| ki9cmf8g38snbanqgpmk12rdm6o707pp | root | root@localhost | 2023-12-28 19:30:38 | Transferring Aleksander Danlos to the Quality Management department |
| c90nt7safn73ehshikgcei65n31kdp7r | root | root@localhost | 2023-12-28 19:29:21 | Adding new employee: Franklen Famemo |
| 9d03mdutiq4u2vl47jh4hgbu97gjgjis | root | root@localhost | 2023-12-28 19:18:03 | Transferring Sajjad Willoner to the Quality Management department |
| fn4hq4ce9a3ctaulji9ltmo8aqb53vk6 | root | root@localhost | 2023-12-28 19:16:41 | Transferring Moriyoshi Merey to the Quality Management department |
| iobdsn9np153r5dljaq2nan2f35cl8t3 | root | root@localhost | 2023-12-28 19:15:32 | Transferring Weiyi Meriste to the Quality Management department |
| len12g9vo7htirc28ut0csmrmpa5lbdt | root | root@localhost | 2023-12-28 18:58:42 | temporarily removing non-engineering employees |
| a3p5ulod2rerno98191lhqnuhhfgppno | timsehn | tim@dolthub.com | 2023-11-27 23:28:41 | imported data |
| oi4sc8120pevl344fv60b865s2p73169 | timsehn | tim@dolthub.com | 2023-11-27 23:15:25 | Initialize data repository |
+----------------------------------+-----------+-----------------+---------------------+---------------------------------------------------------------------+There are several things we should clean up before we open our pull request:
- In commit len12g9vo7htirc28ut0csmrmpa5lbdt, we temporarily deleted all non-engineering employees, since we're only dealing with engineering employees currently. This made it a little easier to explore the data and run some queries, but we definitely can't include these changes in our pull request.
- Commits iobdsn9np153r5dljaq2nan2f35cl8t3, fn4hq4ce9a3ctaulji9ltmo8aqb53vk6, 9d03mdutiq4u2vl47jh4hgbu97gjgjis, and ki9cmf8g38snbanqgpmk12rdm6o707pp are all transferring employees to the Quality Management department. We'd like to combine these four commits into a single commit, since they're all related to the same change.
- We added a new employee in commit c90nt7safn73ehshikgcei65n31kdp7r, but we goofed up the name and had to fix it in commit 1kabuobsb2ouhlcb7se0gitcij90kll5. We'd like to combine those commits and fix the misspelled name in the first commit message.
Fortunately, all of these are super easy to fix with an interactive rebase. Let's get started!
-- Confirm that we're on our development branch. This is the branch that will be rebased and have
-- its commit history rewritten.
select active_branch();
+-----------------+
| active_branch() |
+-----------------+
| eng-updates |
+-----------------+
-- Start an interactive rebase onto the main branch.
call dolt_rebase('--interactive', 'main');
+--------+----------------------------+
| status | message |
+--------+----------------------------+
| 0 | interactive rebase started |
+--------+----------------------------+
-- Examine the default rebase plan.
select * from dolt_rebase order by rebase_order;
+--------------+--------+----------------------------------+---------------------------------------------------------------------+
| rebase_order | action | commit_hash | commit_message |
+--------------+--------+----------------------------------+---------------------------------------------------------------------+
| 1.00 | pick | len12g9vo7htirc28ut0csmrmpa5lbdt | temporarily removing non-engineering employees |
| 2.00 | pick | iobdsn9np153r5dljaq2nan2f35cl8t3 | Transferring Weiyi Meriste to the Quality Management department |
| 3.00 | pick | fn4hq4ce9a3ctaulji9ltmo8aqb53vk6 | Transferring Moriyoshi Merey to the Quality Management department |
| 4.00 | pick | 9d03mdutiq4u2vl47jh4hgbu97gjgjis | Transferring Sajjad Willoner to the Quality Management department |
| 5.00 | pick | c90nt7safn73ehshikgcei65n31kdp7r | Adding new employee: Franklen Famemo |
| 6.00 | pick | ki9cmf8g38snbanqgpmk12rdm6o707pp | Transferring Aleksander Danlos to the Quality Management department |
| 7.00 | pick | 1kabuobsb2ouhlcb7se0gitcij90kll5 | fixing typos in Franklin Fameemo's name |
+--------------+--------+----------------------------------+---------------------------------------------------------------------+
-- Make sure we drop the commit that deleted all non-engineering employees.
update dolt_rebase set action = 'drop' where commit_hash = 'len12g9vo7htirc28ut0csmrmpa5lbdt';
-- Now let's combine the four commits that transfer employees to the Quality Management department.
-- We leave the first commit as 'pick' and change the rest to 'squash', since the squash and
-- fixup actions require a parent commit to squash into. Commit ki9cmf8g38snbanqgpmk12rdm6o707pp
-- also needs to have its rebase_order moved up, so that it gets reordered and squashed together
-- with the other related commits.
update dolt_rebase set action = 'squash' where commit_hash in ('fn4hq4ce9a3ctaulji9ltmo8aqb53vk6','9d03mdutiq4u2vl47jh4hgbu97gjgjis');
update dolt_rebase set rebase_order=4.1, action='squash' where commit_hash='ki9cmf8g38snbanqgpmk12rdm6o707pp';
-- Finally, let's fix Franklin's name in the commit message for commit c90nt7safn73ehshikgcei65n31kdp7r,
-- and set the action for commit 1kabuobsb2ouhlcb7se0gitcij90kll5 to 'fixup'. The fixup
-- action is very similar to squash, except that it does NOT append the commit message to
-- the previous commit message.
update dolt_rebase set action='reword', commit_message='Adding new employee: Franklin Fameemo' where commit_hash='c90nt7safn73ehshikgcei65n31kdp7r';
update dolt_rebase set action='fixup' where commit_hash='1kabuobsb2ouhlcb7se0gitcij90kll5';
-- Now let's double check our rebase plan and make sure it looks correct.
employees> select * from dolt_rebase order by rebase_order;
+--------------+--------+----------------------------------+---------------------------------------------------------------------+
| rebase_order | action | commit_hash | commit_message |
+--------------+--------+----------------------------------+---------------------------------------------------------------------+
| 1.00 | drop | len12g9vo7htirc28ut0csmrmpa5lbdt | temporarily removing non-engineering employees |
| 2.00 | pick | iobdsn9np153r5dljaq2nan2f35cl8t3 | Transferring Weiyi Meriste to the Quality Management department |
| 3.00 | squash | fn4hq4ce9a3ctaulji9ltmo8aqb53vk6 | Transferring Moriyoshi Merey to the Quality Management department |
| 4.00 | squash | 9d03mdutiq4u2vl47jh4hgbu97gjgjis | Transferring Sajjad Willoner to the Quality Management department |
| 4.10 | squash | ki9cmf8g38snbanqgpmk12rdm6o707pp | Transferring Aleksander Danlos to the Quality Management department |
| 5.00 | reword | c90nt7safn73ehshikgcei65n31kdp7r | Adding new employee: Franklin Fameemo |
| 7.00 | fixup | 1kabuobsb2ouhlcb7se0gitcij90kll5 | fixing typos in Franklin Fameemo's name |
+--------------+--------+----------------------------------+---------------------------------------------------------------------+
-- Start rebasing commits now that we've adjusted our rebase plan.
call dolt_rebase('--continue');
+--------+------------------------------+
| status | message |
+--------+------------------------------+
| 0 | interactive rebase completed |
+--------+------------------------------+
-- Finally, let's take a look at our commit history now that the rebase is complete.
employees> select * from dolt_log;
+----------------------------------+-----------+-----------------+---------------------+---------------------------------------------------------------------+
| commit_hash | committer | email | date | message |
+----------------------------------+-----------+-----------------+---------------------+---------------------------------------------------------------------+
| nnf0ealo4l2ot9mpo8d650d3jkcbo4hu | root | root@localhost | 2023-12-28 19:56:37 | Adding new employee: Franklin Fameemo |
| tasavhmupbiet72595mc3vvmms9gpre8 | root | root@localhost | 2023-12-28 19:56:37 | Transferring Weiyi Meriste to the Quality Management department |
| | | | | |
| | | | | Transferring Moriyoshi Merey to the Quality Management department |
| | | | | |
| | | | | Transferring Sajjad Willoner to the Quality Management department |
| | | | | |
| | | | | Transferring Aleksander Danlos to the Quality Management department |
| a3p5ulod2rerno98191lhqnuhhfgppno | timsehn | tim@dolthub.com | 2023-11-27 23:28:41 | imported data |
| oi4sc8120pevl344fv60b865s2p73169 | timsehn | tim@dolthub.com | 2023-11-27 23:15:25 | Initialize data repository |
+----------------------------------+-----------+-----------------+---------------------+---------------------------------------------------------------------+Our commit history is looking clean and ready to be sent out for review in a pull request! There's no trace of the typos we made, or the test changes we made to remove data, and several small commits have been combined into a single commit.
Now when we open our PR on DoltHub, our reviewer will see a very simple, and clean commit history with just two commits.
Rebase Roadmap
This is the first release of rebase functionality in Dolt, and there are still more features and options we want to build out. If you want to use any of these features let us know! We prioritize our work based on what customers tell us they want, so it's a big help to hear from you. 💜
Conflict Resolution – Running a rebase that triggers a conflict currently causes the rebase to be aborted automatically. In the future, we will allow customers to manually resolve any conflicts that occur during an interactive rebase and then continue processing the rebase plan. Until then, if you do hit a conflict, you can 1) try the rebase again with a different commit order that doesn't cause the conflict, or 2) retry the rebase and add in a new commit to the rebase plan that will prevent the conflict from occurring, or 3) skip rebase and use cherry-pick to apply the commits individually while manually resolving any conflicts that occur.
Non-interactive Rebasing – The initial rebase support is focused on an interactive rebase experience, where you get to examine and adjust the rebase plan, then run dolt_rebase('--continue') to execute it. Git rebase also supports a non-interactive experience where the default rebase plan will run without requiring the user to call --continue.
CLI Support – The initial support for rebase is only available through Dolt's SQL interface. CLI support will enable the same rebase functionality from the command line.
Rebase Actions – Dolt supports the following actions in a rebase plan: drop, pick, squash, fixup, and reword. Git supports a few additional rebase actions, such as exec and edit.
Rebase Options – Git supports a LOT of options for rebasing. We've started with what we think are the most important options, and we're happy to add specific options as customers need them. Just drop us a line to let us know!
Merge Commits – By default, rebase in Git and Dolt ignores merge commits. This is because the default use case for rebase is to clean up branches before merging them back to an upstream branch, and at the end of the rebase, the new commits are based off of the tip of the upstream branch, so there often isn't a reason to recreate merge commits. That said, rebase is a powerful feature and can be used for lots of different situations, and you may want to recreate merge commits for some of those more advanced use cases. In 2019, Git 1.28.0 added support for the --rebase-merges option which allows you to recreate merge commit topologies.
Wrap up
We're really excited about Dolt's new support for rebase! This feature has been requested by many customers, and was the largest remaining gap on major Git features that we didn't support in Dolt yet.
If you haven't tried out Dolt, go ahead and install it, test out the new rebase support, and let us know what you think! If you hit any problems or have any feedback, we'd love to hear it! You can send us an issue on GitHub, or swing by our Discord server and chat with us. Our dev team hangs out on Discord every day while we're working and we're always happy to talk about databases and versioning! 🤓