Rust 2020

What the developers at Parity would like to see happen in Rust in 2020.

At Parity we have been on board the Rust train from very early on. Aside from a few specific exceptions, our entire codebase is in Rust - including the Parity Ethereum client, the Substrate blockchain development kit, and the Polkadot network based on Substrate. We are probably one of the biggest Rust users out there and most certainly have one of the largest Rust code bases, which gives us a unique perspective in the ecosystem. And why wouldn't we develop in Rust? Rust is awesome. We wouldn't be able to move this fast and quickly develop new features and blockchain experiments if it weren't for the Rust type system, the borrow checker, and the tooling provided. Rust's fast release cycle is also very important to us. Our Substrate codebase requires latest stable because we often merge code using the latest stabilized features the day a new release is out, and many of us actually develop on nightly. Kudos to all the people that make these fast turnarounds happen! It is truly amazing.

While some might think such a fast release cycle sacrifices stability or quality, we have not had that experience. Nor have we experienced that this leads to feature creep or other common side effects. Quite the opposite: overall, we think Rust is moving in the right direction. For example, async/await, although not vital for our code base, is a great addition that makes the code a lot more readable. We are very happy to see this released, and to see the entire async story continue to move along.

That said, with a codebase as big as ours, we see some unique challenges ahead that we'd like to bring attention to: some bugs and language feature requests, some ergonomics and security around the compiler, and governance overall.

Bugs and features requests

Even small bugs can add up quite quickly in a huge codebase. A pretty problematic one for us is the feature leakage of cargo: We have a lot of crates, and to make integration testing easier, we often include bigger setups in the dev dependencies--which more often than not leads to annoying circular dependencies. Similarly, we are building (part of) our project for Wasm and native at same time, but because that would leak through std, we have to resort to some n...ifty hacks that we'd rather not do.

Similarly, on the language side of things, Substrate is a mostly generic blockchain development framework--generic over the encryption and database you use. That is made possible through the great traits framework Rust provides. However, here too we are coming across some limitations, many of which we work around but would prefer not to.

For example, adding a 'where' clause to a trait declaration forces us to replicate this where bound whenever we want to bound a type to this trait. Or we need to replicate bounds on an associated type through the codebase. We are confident that most of these trait resolutions problems will be solved by switching to chalk in rustc. We are also eagerly looking forward to Specialization in Rust. Currently Substrate is using a lot of so-called "macro-magic" to prevent users from writing the same code over and over again. However, a lot of the users are not happy that we rely so much on macros, as it hides the actual implementation. We believe that we could reduce the macro usage by just heavily simplifying macros when Rust ships Specialization.

Compiler ergonomics and security
Aside from specific bugs and features, an important aspect is compiler ergonomics and security. We recognize and highly appreciate the work that has been put into performance optimization and compile time improvements, but here, too, we have to resort to rather ugly in-between solutions. With a codebase that pulls in over 1000 crates now and goes over the 160kLOC, even minor changes easily mean eight minute compile time on "the beast." The beast is our internal build server we cargo remote into to build, because it would easily take twice as long on our otherwise really beefy laptop hardware. Even with incremental builds and sccache on, a code-try-repeat-cycle is easily minutes per iteration. Other tooling intended to make developers' lives easier, like Rust analyzer and RLS, continuously choke on our project.

This situation gets even worse when we attempt to get people to use Substrate. Substrate is a development kit. Rather than a dylib or binary, we have to ship the entire source code and build it all on the user's computer so they can build their library. Because Rust isn't a widely available development environment yet, our main getting started tutorial contains a script to set all this up--which itself takes 45 minutes on a modern system. While optimizations in the past have mostly focused on subsequent builds, the first setup and build time is still very high and a real pain point to get people to even start looking at Rust. When at long last we get people to complete the setup, they tell us that Rust IS AWESOME and a perfect fit for exactly what we are building.

Aside from compiler ergonomics, security is of utmost importance to us. Our project runs decentralized peer-to-peer networks in untrusted environments. One key feature is allowing the authorities of said network to upgrade their business logic through providing a new Wasm-compiled binary. We highly appreciate the efforts being made getting a certified compiler going. However, what is of higher concern for our use case is deterministic builds. Clearly, you can't expect everyone to introspect the Wasm blob and figure out whether it does what it is supposed to do. However, if you could provide the source code and when someone builds it (with the same compiler version), they would come up with the binary-exact same copy, then they could just review the source code and trust that. This, unfortunately, is currently not the case, even for non-std-wasm.

We think this could also be a part-way solution to our first-build problem: if we had deterministic builds, we could share build caches across system and perhaps even ship pre-populated assets rather than always having to build locally--at least for all things pure Rust. We were excited to notice that people have resumed working on deterministic builds, and we're eager to see what their work yields in 2020.

Governance: project and expectation management
For many of the issues raised above, we are also happy to jump in and help out--and on other issues as well. We are a Rust company after all---we believe in the language, its ecosystem and the community, and want to be a valuable participant in it. Beyond that, the stability and its continued progress is vital for our business. Thus all contributions to Rust core, std and its direct tooling are championed inside Parity, even though that isn't very easy when you are not a full-time contributor.

We, too, have team members who are interested in helping on specialization or fixing the aforementioned bugs. However, it's often unclear whether the work is worthwhile. To a business, it is hard to argue that one might spend a month or two working on a new feature without any assurance that the approach taken would be accepted. Personally, I think it might even be more frustrating if one's part-time volunteer work might be thrown out because it wasn't acceptable to begin with. Yet, there still is no clear path to get some official agreement, only personal assurances--but as a company that champions transparent governance, we believe there must be a better way.

One example of how this unclear decision-making process takes a toll is the async-await-discussion. While it eventually was resolved in an elegant and satisfying way, the emotional toll the discussion had on participants suggests there may be a better way. There is also the bystander effect: the potential of facing such an intense and difficult crowd may cause contributors to hesitate. In particular, non-native English speakers may not feel adept at fluently defending their positions, or contributors may not have the time to spend participating in discussions.

Of course, this is a challenge that many volunteer projects share: with growing contributions comes growing communication, and at some point all the core team seems to be doing is communicating and coordinating, which may not be what that team enjoys. We are concerned to see people leave the community over this and related issues. We cannot build a sustainable community if it burns through its people!

When it comes to wondering how, we, as a company, could help or contribute resources to resolve some of those structural and larger issues, it comes down to speaking to individuals and specific teams rather than transparently in the open. Parity is in the distributed, collaborative governance business; we value transparency and pragmatism. We do not want any one entity to have full control over Rust---including us---but would prefer it to be a participatory, inclusive, transparent and democratic process.

We believe 2020 will show increased interest by (major) companies to be involved in Rust and thus are concerned that the lack of an official way to participate could encourage companies to use personal relationships and "buying" seats at the table through hiring highly influential people. We highly appreciate the forming of the partnership and governance working groups (this article's primary author is, outside of work, a part-time member of the governance working group) and are keen to see what comes of these groups and how we could be of assistance.

Overall, we are looking forward to Rust in 2020. We are more enthusiastic about Rust than ever, and appreciate the open discussions in the Rust community. We most definitely picked the right language and community to bet on.