A Data Mesh from Scratch in Rust — Part 8 —Server Design
Now we finally, move into designing the database server, which a part of this series. Before I get into the implementation, I wanted to use one post, again, to describe the system design. I think it will make it a bit easier to follow what’s going on. It was a lot of blood and sweat and after a couple of days, I started taking the path of least resistance. So any feedback is welcome on these posts.
The first thing we do before even getting to the server is to figure out the interface between the server and the storage engine. The main idea here, obviously, was to expose the relevant operations to add, delete and read events. I decided to use protobufs for data serialization. But overall this was about combining the different APIs in the different data structures to get the desired effect — like add, delete and get — from the storage engine. However, there was one big question here: do I add locks and enclose them in Arcs? This, while looking like an implementation problem, actually is a design question — how granular do we want the locking to be! Because the increased level of locking granularity would require careful maneuvering to ensure that deadlocks don’t occur.
Next we create a server. I went back and forth quite a few times between a HTTP based vs a TCP based server. I finally chose to go with the TCP server. There was not much to the decision — I have used actix before to build the reverse proxy before, so I wanted to use tokio to build a TCP server. I also got the idea to separate out the user interface into a server and cli from the NaiveKV blog. So that gave me the opportunity to work with clap as well.
When we get to the server though, I had to use locks and Arc. I hate it and I tried but there is just no way around that I could find. This is something I miss about not programming this in C. In C, we could add a lock in or two in our server and decide for the granularity ourselves. Rust sort of forces us on this. The other thing I miss in Rust, from C, is the ability to build a lock-free buffer, maybe it’s more apt to say the requirement is for a wait-free one. In C, you can build a queue with the consumption end free from locks — you still need locks and condition variables on the producer side once there are more than one producer. Though, to be fair, I do not know if Rust’s VecDequeu does anything similar internally.
Anyhow, given those choices it was I think the better choice to use a global read-write lock on the entire interface so as to avoid all deadlock issues. I also tried to separate out the writes and the reads. But, unfortunately, they are not as clean as they could be. The whole thing definitely needs to be optimized for scalability.
In terms of compaction, both SSTables and MemTables, a separate background thread gets informed when events are added. MemTables are compacted when they are beyond a certain size. SSTables are compacted at the same time, though a separate policy for that is certainly possible.
Finally, for the client, we have a clap based cli that sends Requests, protobuf messages, to the Server and receives Response messages in response. This client uses String payloads but any other payload is certainly possible.
Next we will start talking about the server design while throwing in details about the interface as we go.
