EuAndreh's website

Durable persistent trees and parser combinators - building a database

2020-11-12T00:00:00+00:00

I’ve received with certain frequency messages from people wanting to know if I’ve made any progress on the database project I’ve written about.

There are a few areas where I’ve made progress, and here’s a public post on it.

Proof-of-concept: DAG log

The main thing I wanted to validate with a concrete implementation was the concept of modeling a DAG on a sequence of datoms.

The notion of a datom is a rip-off from Datomic, which models data with time aware facts, which come from RDF. RDF’s fact is a triple of subject-predicate-object, and Datomic’s datoms add a time component to it: subject-predicate-object-time, A.K.A. entity-attribute-value-transaction:

1
2
3

[[person :likes "pizza" 0 true]
 [person :likes "bread" 1 true]
 [person :likes "pizza" 1 false]]

plaintext

The above datoms say: - at time 0, person like pizza; - at time 1, person stopped liking pizza, and started to like bread.

Datomic ensures total consistency of this ever growing log by having a single writer, the transactor, that will enforce it when writing.

In order to support disconnected clients, I needed a way to allow multiple writers, and I chose to do it by making the log not a list, but a directed acyclic graph (DAG):

[[person :likes  "pizza"  0 true]
 [0      :parent :db/root 0 true]
 [person :likes  "bread"  1 true]
 [person :likes  "pizza"  1 false]
 [1      :parent 0        1 true]]

plaintext

The extra datoms above add more information to build the directionality to the log, and instead of a single consistent log, the DAG could have multiple leaves that coexist, much like how different Git branches can have different "latest" commits.

In order to validate this idea, I started with a Clojure implementation. The goal was not to write the actual final code, but to make a proof-of-concept that would allow me to test and stretch the idea itself.

This code already exists, but is yet fairly incomplete:

the building of the index isn’t done yet (with some commented code on the next step to be implemented)
the indexing is extremely inefficient, with more than one occurrence of O² functions;
no query support yet.

Top-down and bottom-up

However, as time passed and I started looking at what the final implementation would look like, I started to consider keeping the PoC around.

The top-down approach (Clojure PoC) was in fact helping guide me with the bottom-up, and I now have "promoted" the Clojure PoC into a "reference implementation". It should now be a finished implementation that says what the expected behaviour is, and the actual code should match the behaviour.

The good thing about a reference implementation is that it has no performance of resources boundary, so if it ends up being 1000× slower and using 500× more memory, it should be find. The code can be also 10× or 100× simpler, too.

Top-down: durable persistent trees

In promoting the PoC into a reference implementation, this top-down approach now needs to go beyond doing everything in memory, and the index data structure now needs to be disk-based.

Roughly speaking, most storage engines out there are based either on B-Trees or LSM Trees, or some variations of those.

But when building an immutable database, update-in-place B-Trees aren’t an option, as it doesn’t accommodate keeping historical views of the tree. LSM Trees may seem a better alternative, but duplication on the files with compaction are also ways to delete old data which is indeed useful for a historical view.

I think the thing I’m after is a mix of a Copy-on-Write B-Tree, which would keep historical versions with the write IO cost amortization of memtables of LSM Trees. I don’t know of any B-Tree variant out there that resembles this, so I’ll call it "Flushing Copy-on-Write B-Tree".

I haven’t written any code for this yet, so all I have is a high-level view of what it will look like:

like Copy-on-Write B-Trees, changing a leaf involves creating a new leaf and building a new path from root to the leaf. The upside is that writes a lock free, and no coordination is needed between readers and writers, ever;
the downside is that a single leaf update means at least H new nodes that will have to be flushed to disk, where H is the height of the tree. To avoid that, the writer creates these nodes exclusively on the in-memory memtable, to avoid flushing to disk on every leaf update;
a background job will consolidate the memtable data every time it hits X MB, and persist it to disk, amortizing the cost of the Copy-on-Write B-Tree;
readers than will have the extra job of getting the latest relevant disk-resident value and merge it with the memtable data.

The key difference to existing Copy-on-Write B-Trees is that the new trees are only periodically written to disk, and the intermediate values are kept in memory. Since no node is ever updated, the page utilization is maximum as it doesn’t need to keep space for future inserts and updates.

And the key difference to existing LSM Trees is that no compaction is run: intermediate values are still relevant as the database grows. So this leaves out tombstones and value duplication done for write performance.

One can delete intermediate index values to reclaim space, but no data is lost on the process, only old B-Tree values. And if the database ever comes back to that point (like when doing a historical query), the B-Tree will have to be rebuilt from a previous value. After all, the database is a set of datoms, and everything else is just derived data.

Right now I’m still reading about other data structures that storage engines use, and I’ll start implementing the "Flushing Copy-on-Write B-Tree" as I learn morefootnote:learn-more-db[ If you are interested in learning more about this too, the very best two resources on this subject are Andy Pavlo’s "Intro to Database Systems" course and Alex Petrov’s "Database Internals" book. ] and mature it more.

Bottom-up: parser combinators and FFI

I chose Rust as it has the best WebAssembly tooling support.

My goal is not to build a Rust database, but a database that happens to be in Rust. In order to reach client platforms, the primary API is the FFI one.

I’m not very happy with current tools for exposing Rust code via FFI to the external world: they either mix C with C++, which I don’t want to do, or provide no access to the intermediate representation of the FFI, which would be useful for generating binding for any language that speaks FFI.

I like better the path that the author of cbindgen crate proposes: emitting an data representation of the Rust C API (the author calls is a ffi.json file), and than building transformers from the data representation to the target language. This way you could generate a C API and the node-ffi bindings for JavaScript automatically from the Rust code.

So the first thing to be done before moving on is an FFI exporter that doesn’t mix C and C++, and generates said ffi.json, and than build a few transformers that take this ffi.json and generate the language bindings, be it C, C++, JavaScript, TypeScript, Kotlin, Swift, Dart, _etc_footnote:ffi-langs[ Those are, specifically, the languages I’m more interested on. My goal is supporting client applications, and those languages are the most relevant for doing so: C for GTK, C++ for Qt, JavaScript and TypeScript for Node.js and browser, Kotlin for Android and Swing, Swift for iOS, and Dart for Flutter. ].

I think the best way to get there is by taking the existing code for cbindgen, which uses the syn crate to parse the Rust codefootnote:rust-syn[ The fact that syn is an external crate to the Rust compiler points to a big warning: procedural macros are not first class in Rust. They are just like Babel plugins in JavaScript land, with the extra shortcoming that there is no specification for the Rust syntax, unlike JavaScript.

As flawed as this may be, it seems to be generally acceptable and adopted, which works against building a solid ecosystem for Rust.

The alternative that rust-ffi implements relies on internals of the Rust compiler, which isn’t actually worst, just less common and less accepted. ], and adapt it to emit the metadata.

I’ve started a fork of cbindgen: x-bindgenfootnote:x-bindgen[ EDIT: now archived, the experimentation was fun. I’ve started to move more towards C, so this effort became deprecated. ]. Right now it is just a copy of cbindgen verbatim, and I plan to remove all C and C++ emitting code from it, and add a IR emitting code instead.

When starting working on x-bindgen, I realized I didn’t know what to look for in a header file, as I haven’t written any C code in many years. So as I was writing libedn, I didn’t know how to build a good C API to expose. So I tried porting the code to C, and right now I’m working on building a good C API for a JSON parser using parser combinators: ParsecCfootnote:parsecc[ EDIT: now also archived. ].

After "finishing" ParsecC I’ll have a good notion of what a good C API is, and I’ll have a better direction towards how to expose code from libedn to other languages, and work on x-bindgen then.

What both libedn and ParsecC are missing right now are proper error reporting, and property-based testing for libedn.

Conclusion

I’ve learned a lot already, and I feel the journey I’m on is worth going through.

If any of those topics interest you, message me to discuss more or contribute! Patches welcome!

The database I wish I had

2020-08-31T00:00:00+00:00

I watched the talk "Platform as a Reflection of Values: Joyent, Node.js and beyond" by Bryan Cantrill, and I think he was able to put into words something I already felt for some time: if there’s no piece of software out there that reflects your values, it’s time for you to build that softwarefootnote:talk-time[ At the very end, at time 29:49. When talking about the draft of this article with a friend, he noted that Bryan O’Sullivan (a different Bryan) says a similar thing on his talk "Running a startup on Haskell", at time 4:15. ].

I kind of agree with what he said, because this is already happening to me. I long for a database with a certain set of values, and for a few years I was just waiting for someone to finally write it. After watching his talk, Bryan is saying to me: "time to stop waiting, and start writing it yourself".

So let me try to give an overview of such database, and go over its values.

Overview

I want a database that allows me to create decentralized client-side applications that can sync data.

The best one-line description I can give right now is:

It’s sort of like PouchDB, Git, Datomic, SQLite and Mentat.

A more descriptive version could be:

An embedded, immutable, syncable relational database.

Let’s go over what I mean by each of those aspects one by one.

Embedded

I think the server-side database landscape is diverse and mature enough for my needs (even though I end up choosing SQLite most of the time), and what I’m after is a database to be embedded on client-side applications itself, be it desktop, browser, mobile, etc.

The purpose of such database is not to keep some local cache of data in case of lost connectivity: we have good solutions for that already. It should serve as the source of truth, and allow the application to work on top of it.

SQLite is a great example of that: it is a very powerful relational database that runs almost anywhere. What I miss from it that SQLite doesn’t provide is the ability to run it on the browser: even though you could compile it to WebAssembly, it assumes a POSIX filesystem that would have to be emulatedfootnote:posix-sqlite[ It was pointed out to me that SQLite doesn’t assume the existence of a POSIX filesystem, as I wrongly stated. Thanks for the correction.

This makes me consider it as a storage backend all by itself. I initially considered having an SQLite storage backend as one implementation of the POSIX filesystem storage API that I mentioned. My goal was to rely on it so I could validate the correctness of the actual implementation, given SQLite’s robustness.

However it may even better to just use SQLite, and get an ACID backend without recreating a big part of SQLite from scratch. In fact, both Datomic and PouchDB didn’t create an storage backend for themselves, they just plugged on what already existed and already worked. I’m beginning to think that it would be wiser to just do the same, and drop entirely the from scratch implementation that I mentioned.

That’s not to say that adding an IndexedDB compatibility layer to SQLite would be enough to make it fit the other requirements I mention on this page. SQLite still is an implementation of a update-in-place, SQL, table-oriented database. It is probably true that cherry-picking the relevant parts of SQLite (like storage access, consistency, crash recovery, parser generator, etc.) and leaving out the unwanted parts (SQL, tables, threading, etc.) would be better than including the full SQLite stack, that’s simply an optimization. Both could even coexist, if desired.

SQLite would have to be treated similarly to how Datomic treats SQL databases: instead of having a table for each entities, spread attributes over the tables, etc., it treats SQL databases as a key-value storage so it doesn’t have to re-implement interacting with the disk that other databases do well.

The tables would contain blocks of binary data, so there isn’t a difference on how the SQLite storage backend behaves and how the IndexedDB storage backend behaves, much like how Datomic works the same regardless of the storage backend, same for PouchDB.

I welcome corrections on what I said above, too. ].

PouchDB is another great example: it’s a full reimplementation of CouchDB that targets JavaScript environments, mainly the browser and Node.js. However I want a tool that can be deployed anywhere, and not limit its applications to places that already have a JavaScript runtime environment, or force the developer to bundle a JavaScript runtime environment with their application. This is true for GTK+ applications, command line programs, Android apps, etc.

Mentat was an interesting project, but its reliance on SQLite makes it inherit most of the downsides (and benefits too) of SQLite itself.

Having such a requirement imposes a different approach to storage: we have to decouple the knowledge about the intricacies of storage from the usage of storage itself, so that a module (say query processing) can access storage through an API without needing to know about its implementation. This allows the database to target a POSIX filesystems storage API and an IndexedDB storage API, and make the rest of the code agnostic about storage. PouchDB has such mechanism (called adapters) and Datomic has them too (called storage services).

This would allow the database to adapt to where it is embedded: when targeting the browser the IndexedDB storage API would provide the persistence layer that the database requires, and similarly the POSIX filesystem storage API would provide the persistence layer when targeting POSIX systems (like desktops, mobile, etc.).

But there’s also an extra restriction that comes from by being embedded: it needs to provide and embeddable artifact, most likely a binary library object that exposes a C compatible FFI, similar to how SQLite does. Bundling a full runtime environment is possible, but doesn’t make it a compelling solution for embedding. This rules out most languages, and leaves us with C, Rust, Zig, and similar options that can target POSIX systems and WebAssembly.

Immutable

Being immutable means that only new information is added, no in-place update ever happens, and nothing is ever deleted.

Having an immutable database presents us with similar trade-offs found in persistent data structures, like lack of coordination when doing reads, caches being always coherent, and more usage of space.

Datomic is the go to database example of this: it will only add information (datoms) and allows you to query them in a multitude of ways. Stuart Halloway calls it "accumulate-only" over "append-only"footnote:accumulate-only[ Video "Day of Datomic Part 2" on Datomic’s information model, at time 12:28. ]:

It’s accumulate-only, it is not append-only. So append-only, most people when they say that they’re implying something physical about what happens.

Also a database can be append-only and overwrite existing information with new information, by doing clean-ups of "stale" data. I prefer to adopt the "accumulate-only" naming and approach.

Git is another example of this: new commits are always added on top of the previous data, and it grows by adding commits instead of replacing existing ones.

Git repositories can only grow in size, and that is not only an acceptable condition, but also one of the reasons to use it.

All this means that no in-place updates happens on data, and the database will be much more concerned about how compact and efficiently it stores data than how fast it does writes to disk. Being embedded, the storage limitation is either a) how much storage the device has or b) how much storage was designed for the application to consume. So even though the database could theoretically operate with hundreds of TBs, a browser page or mobile application wouldn’t have access to this amount of storage. SQLite even says that it does support approximately 280 TBs of data, but those limits are untested.

The upside of keeping everything is that you can have historical views of your data, which is very powerful. This also means that applications should turn this off when not relevantfootnote:no-history[ Similar to Datomic’s :db/noHistory. ].

Syncable

This is a frequent topic when talking about offline-first solutions. When building applications that:

can fully work offline,
stores data,
propagates that data to other application instances,

then you’ll need a conflict resolution strategy to handle all the situations where different application instances disagree. Those application instances could be a desktop and a browser version of the same application, or the same mobile app in different devices.

A three-way merge seems to be the best approach, on top of which you could add application specific conflict resolution functions, like:

pick the change with higher timestamp;
if one change is a delete, pick it;
present the diff on the screen and allow the user to merge them.

Some databases try to make this "easy", by choosing a strategy for you, but I’ve found that different applications require different conflict resolution strategies. Instead, the database should leave this up to the user to decide, and provide tools for them to do it.

Three-way merges in version control are the best example, performing automatic merges when possible and asking the user to resolve conflicts when they appear.

The unit of conflict for a version control system is a line of text. The database equivalent would probably be a single attribute, not a full entity or a full row.

Making all the conflict resolution logic be local should allow the database to have encrypted remotes similar to how git-remote-gcrypt adds this functionality to Git. This would enable users to sync the application data across devices using an untrusted intermediary.

Relational

I want the power of relational queries on the client applications.

Most of the arguments against traditional table-oriented relational databases are related to write performance, but those don’t apply here. The bottlenecks for client applications usually aren’t write throughput. Nobody is interested in differentiating between 1 MB/s or 10 MB/s when you’re limited to 500 MB total.

The relational model of the database could either be based on SQL and tables like in SQLite, or maybe datalog and datoms like in Datomic.

From aspects to values

Now let’s try to translate the aspects above into values, as suggested by Bryan Cantrill.

Portability

Being able to target so many different platforms is a bold goal, and the embedded nature of the database demands portability to be a core value.

Integrity

When the local database becomes the source of truth of the application, it must provide consistency guarantees that enables applications to rely on it.

Expressiveness

The database should empower applications to slice and dice the data in any way it wants to.

Next steps

Since I can’t find any database that fits these requirements, I’ve finally come to terms with doing it myself.

It’s probably going to take me a few years to do it, and making it portable between POSIX and IndexedDB will probably be the biggest challenge. I got myself a few books on databases to start.

I wonder if I’ll ever be able to get this done.

External links

See discussions on Reddit, lobsters, HN and a lengthy email exchange.