This is a small post in appreciation of Rust, the programming language, and its ecosystem.
If you’ve read my other posts you’ll know that I have a strong interest in Rust. I have been using Rust for quite a number of years and I have released several posts about it, for example the performance of Rust when compiled to WASM. I’ve also built some software using Rust: Lishwist, Syswall and the Rust/WASM graphics library.
I have a confession to make though… whenever I consider writing something in Rust I am suddenly overcome with a vision of my near future. That future is filled with complex borrow checker errors, and a lot of time spent getting the program to compile. That does not always happen of course, but it’s just the way I imagine most Rust projects going.
I also write a lot of Python. I find Python to be the closest analogue to the pseudo-code that I write in my head. The speed that I can transform my head-code to real code is fastest when using Python. I know that if I have a problem then solving it with Python will be relatively quick and easy for me compared to other languages. The solution might not always be right first time, but I can quickly test and hone the solution until it is acceptable.
Because of this I end up writing a lot of “throw-away” scripts in Python. This includes work that I do for clients; if there’s a one-off complex job that I need the computer to perform then Python will usually be my tool of choice.
Today I needed to process a large number of records for a client. I was required to compare each record with every other and for each couple to calculate a similarity score. This computation required a comparison between seven string fields on both records, and required a squaring and division per field too. While writing the script in Python was very quick and easy I was already uneasy about the performance. Sure enough, when I ran the script I found that it was able to perform all of the comparisons for a single record at a rate of approximately one per second. There were approximately 1.1 million records though, meaning that the script would take almost 13 days to finish running.
A quick side note: the fact that CPython can compare two records across seven different fields with those arithmetic operations thrown in too at the rate of ~1 million iterations per second is very impressive! It just shows how fast CPUs are these days!
Despite how impressive CPUs and CPython are, that execution time was not acceptable. I went back to my mental pseudo-code and checked to see if the algorithm could be optimised, however I could not find a way to do so. My next step was to utilise either multi-threading or multi-processing in Python.
The multi-processing attempt failed at the first step. I wanted to share the large in-memory list between processes but as far as I could tell it was only possible for me to share arrays of integers or floats. I tried multi-threading but found that the performance was about the same as with the single threaded version. My theory for this is that this is down to Python’s GIL: as the record list is shared between threads the GIL prevents concurrent reads from occurring. I might be wrong with this theory, but my mind was already working its way towards Rust…
At this point I was in a quandary. I knew that I could communicate read-only concurrent access to a Vec via Rust’s type system which should mean that I could get much better performance by utilising more CPU cores. However my usual vision of a sea of complicated borrow checker errors filling the next few hours of my life yet again haunted me. I convinced myself that writing this in Rust would lead to a solution with a much higher performance, but the price of this would be more time spent in development, “battling with the borrow checker” as they say.
I could not have been more wrong.
I started by using the csv crate to read the CSV file containing the records and the serde crate to de-serialise them into a vector of structs. I made a couple of mistakes during the mapping of results when reading each row but the compiler always told me clearly what was wrong and I was able to fix them easily.
I wrote a function to compare two records and to return a score. I then wrote another function which used two nested iterators to compare each record to every other record. Very quickly I had a program which compiled and was ready to run.
Building in release mode produced a program that generated the same results as Python did, but a little faster. Instead of around 1 iteration per second I was getting about 1.5 iterations per second. Not a great improvement, but I already knew what I wanted to do next: multi-threading!
Multi-threading: the programming model that is notoriously difficult to get right. I took a deep breath and started work. I knew this wouldn’t be simple, but I was confident that Rust’s type system and borrow checker would hold my hand and lead me to a correct solution, even if it might take me a few hours to get there.
At that point I paused. I thought to myself, “why not try Rayon?” I thought it would be worth trying but probably wouldn’t work. I thought that in the process of trying, I’d learn something valuable which would help me to find a proper solution. I added Rayon, changed the Vec’s into_iter() to an into_par_iter() and ran the program again… it maxed out all 24 of my CPU cores and started churning through the list around 20 times faster than the previous solution!
At this point I took a step back. At no point of this process did I get stuck. I translated my mental model of the program to Rust code and quickly fixed a few errors that I made with the help of the compiler. I then changed one function call to allow the program to utilise all available CPU cores and everything just worked!
This experience forever changed the vision that I get when thinking about Rust. Yes, perhaps I could have optimised my algorithm. Perhaps I could have found a way of sharing the user list between processes in Python. Perhaps I could have communicated to Python that shared access to the list across threads was safe. But the point is that I just rewrote it in Rust with minimal hassle and got exactly what I wanted.
So, I may not be a superhero but I can certainly list “Rust” as a superpower!