Dolt is a MySQL-compatible version-controlled database. It's the only SQL database with branches that you can diff, merge and sync. For the last year, we've been rewriting our storage engine to improve Dolt's performance and stability as a production database. We've been shouting about this a lot recently, but it's worth talking about, the results are substantial. Compared to the old storage engine, Dolt is now 3x faster. Today we're announcing a general release of the new storage engine with Dolt v0.50.0. Each new database created with dolt init or CREATE DATABASE ... with use the new storage engine!

A New Foundation

Dolt's novel storage structure is what sets it apart from other databases. Data in Dolt is indexed using Prolly Trees, a B-Tree like data structure introduced by Noms. Dolt used Noms as its original data store to take advantage of the unique capabilities of Prolly Trees. The internal nodes of a Prolly Tree form a Merkel DAG, enabling faster diff operations than any other database. Building on top of Noms was vital to our early success and gave us the ability to quickly prototype our product. However, as we matured as a database it became clear that we needed a faster storage engine if we were going to compete with MySQL on a performance basis. With this goal in mind, we set out to design a new binary format and indexing system that would accelerate Dolt from the ground up.

Making an Impact

We mentioned the new format is fast. Let's take a look at our Sysbench benchmarks to get a sense of just how much better it is:

|       Read Tests      | MySQL | Old Dolt | New Dolt | Change |
|-----------------------|-------|----------|----------|--------|
| covering_index_scan   |  1.96 |     6.55 |     2.66 | - 59 % |
| groupby_scan          | 12.52 |    22.28 |    16.71 | - 25 % |
| index_join            |  1.16 |    16.71 |     4.49 | - 73 % |
| index_join_scan       |  1.12 |    16.12 |     3.82 | - 76 % |
| index_scan            | 30.26 |    71.83 |    53.85 | - 25 % |
| oltp_point_select     |  0.15 |     0.58 |     0.48 | - 17 % |
| oltp_read_only        |  2.97 |     9.91 |     8.58 | - 13 % |
| select_random_points  |   0.3 |     1.39 |     0.74 | - 47 % |
| select_random_ranges  |  0.35 |     1.39 |     1.12 | - 19 % |
| table_scan            | 30.81 |    69.29 |    62.19 | - 10 % |
| types_table_scan      | 69.29 |   215.44 |   183.21 | - 15 % |

|       Write Tests     | MySQL | Old Dolt | New Dolt | Change |
|-----------------------|-------|----------|----------|--------|
| bulk_insert           | 0.001 |    0.001 |    0.001 |    0 % |
| oltp_delete_insert    |  2.97 |    19.65 |    10.84 | - 45 % |
| oltp_insert           |  1.52 |     7.98 |     2.81 | - 65 % |
| oltp_read_write       |  5.67 |    37.56 |    17.01 | - 55 % |
| oltp_update_index     |  1.86 |     9.39 |     4.91 | - 48 % |
| oltp_update_non_index |  1.73 |     6.55 |     5.18 | - 21 % |
| oltp_write_only       |  2.61 |     26.2 |     8.28 | - 68 % |
| types_delete_insert   |  3.13 |    155.8 |    12.08 | - 92 % |

On our join benchmarks, Dolt's new storage engine is 4x faster than the old! And the improvements get even better when we look at more realistic workloads. We recently added TPC-C to our suite of nightly benchmarks with the aim of measuring Dolt's transactional throughput. Dolt's old storage engine manages a little more than 1 transaction per second:

SQL statistics:
        queries performed:
        read:                            208
        write:                           227
        other:                           28
        total:                           463
    transactions:                        14     (1.13 per sec.)
    queries:                             463    (37.40 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

The new storage engine is dramatically faster at 25 transactions per second!

SQL statistics:
        queries performed:
        read:                            3520
        write:                           3661
        other:                           538
        total:                           7719
    transactions:                        256    (25.19 per sec.)
    queries:                             7719   (759.48 per sec.)
    ignored errors:                      15     (1.48 per sec.)
    reconnects:                          0      (0.00 per sec.)

Looking Forward

Shipping Dolt's new storage engine is a major milestone for the product and represents a substantial improvement Dolt's users. Ultimately, our goal is to reach performance parity with MySQL. As we've said before, we don't think our users should choose between Dolt's version control features and MySQL's performance. They should get both. The general release of the new storage engine also means the new binary serialization format is stable as well. As Dolt moves towards 1.0, it will provide a solid foundation for future development work. We're very excited about this new release, and we believe it will have a major impact for our users. If you'd like to learn more about Dolt, or discuss your use case, hit us up on Discord.