A Data Mesh from Scratch in Rust — Part 5— Write Ahead Log

You might want to catch up a little on this series before reading ahead. So far we have talked about defining events and actions and implementing the MemTable and the SSTable.

But what happens when a node crashes. The data in the MemTable is lost in its entirety. This is what the write ahead log (WAL) protects against. The same events that are written to the MemTable are also written to the WAL, which is on-disk, but without the sorting. In case of a crash, one can simply read the WAL back to recover the MemTable.

So without further ado, here is the WAL struct:

WAL

Simple enough! A file path and a buffered writer created from the file path. And as each event arrives, they are promptly recorded in the file (with an epoch which is not stored here but could be).

Then comes the mandatory iterator implementation.

WAL Iterator

Incredible how simple this is, given the time I spent figuring this one out (this is where I mostly figured out the iterator implementation and during MemTable). You are free to look up the code for the rest of the fluff.

Now let’s talk about actually recovering a MemTable from the WAL which is done with an empty struct struct Recovery {} with a single (not counting the helper) recover() methods.

Recovery

That’s it! That is how the entire recovery works:

1. The recover method looks up all the WAL files surviving in the system.
2. Sorts them by their epoch.
3. And then just walks through them, adding the events into a new MemTable before returning said MemTable.

This basically completes our storage engine. The next articles on storage engine are on the bloom filter and error handling. Or you can skip over directly to the server design article without much loss of continuity.