Variable Sized File IO in Rust
Generally buffered IO operations might be a good idea since it allows programs to deal with exactly the amount of data they want. However, the buffered IO tools in Rust — namely, std::io::BufReader and std::io::BufWriter (the ones I am focused on) — also provide pretty easy serialization and deserialization. This blog is, hopefully, a short one providing a useful demonstration.
In a current project, I have a definition of an Event that takes a variable sized Payload where type Payload = Option<Vec<u8>>. Now I wanted to write these event onto a file and then build an iterator to read from it. My first attempt was mostly to understand the process, rather than build the iterator. So here is what I wrote:
Note this part:
Figuring out this serialization is complex, time-consuming and error-prone.
This is where BufReader and BufWriter helps.
In this case, there is no requirement for custom deserialization. All we needed to do is add a delimiter between variable sized segments.
