A Data Mesh from Scratch in Rust — Part 9 — The Interface
This is the implementation story following the design post as part of the data mesh series. We start by defining the serializations using protobuf.
It’s simply defining the request response structure with an operation and status enum thrown in. And that’s all!
This required the following build script:
And module definition:
This builds the proto module in the target directory and the mod file includes it from there. One can replace cargo_out_dir with out_dir to build the module in situ (that will replace the mod.rs file with the generated one). In this case, the serialization methods are available from rdeebee::wire_format module.
The library interface then looks like this:
We store a couple of parameters like the threshold size which will trigger MemTable compaction and the directory where our files will be stored. As I mentioned in the design article, none of them are enclosed with a lock and Arc. All SSTables are stored in a vector as is the bloom filter. The key_to_id_map holds the mapping between the keys used by users to send Requests (protobuf) to the Uuids used by the storage engine internally.
The methods in the interface are simple in that they combine internal APIs to provide functionalities from the library. Operations that take a Request return a Response (protobuf) which indicates success or failure. Others that are about the state of the storage engine itself — like compaction — return Result<(), StorageEngineError>. Finally, the request-response methods also convert Requests to Events.
Please feel free to look at the code for further details. Finally, I have a method called get_stream_by_key in there that finds all events corresponding to a key (get_event_by_key returns the latest event for that key) and returns them in a vector. This is generally a bad idea since that may or may not fit into memory creating pressure on the system. This is mostly a placeholder and will be changed when we step into the streaming part of the system.
Anyway, next we will talk about the server implementation.
