I feel like the author really overstates how hard it is to make a new crate. Making a cargo.toml is simple and frankly you don’t do it so often as to be a burden.
Encouraging cyclic dependencies on the other hand “because it is easy” seems like a lazy cop out.
There are some efforts for lowering the overhead and would be interesting to explore how we could further do it.
First, we allow sharing `Cargo.toml` fields across the workspace since 1.64.0 [0]. In the upcoming 1.71.0, we extend this so that `cargo new` will automatically inherit fields when creating new `Cargo.toml` files [1].
From a different angle, we are also adding support for single-file packages though this is more meant for binaries [2]
Well, they seem to be cognizant of that, but they have a point: compared to simply creating a folder, it is QUITE a lot more effort. It's like the difference between zero and one.
That having been said, I think my bigger issue with this is that this actually gets to be annoying after you do it, not just while doing it. Go manages external dependencies on the module level, which is a level higher than packages. For Rust, it's crates. So a repo with many crates becomes pretty cumbersome. (And a bit moreso if you're trying to maintain a Nix package for said repo, but that's partly the fault of Nix. Only partly, because I actually think Rust's toolchain makes the job of Nix rather difficult...)
As a moderate Nix user/packager, I feel like the ideal model for ecosystem packaging in Nix is poetry2nix, where the existing lockfile hashes are able to be used directly to create pure Nix builds, with no need to generate code, pre-download anything, or have opaque "deps" blobs (like with Bazel).
As far as I can tell, all or most of these requirements are met by Cargo lockfiles as well, so what is the gap that makes Cargo/Rust difficult to do well in Nix?
The main issue I've come across is that when dealing with multi-crate projects, it is not uncommon to have two versions of the exact same version number of a Rust crate in two different dependencies. Due to a limitation in the machinery, this is painful.
The other thing that's weird is that you have to embed the lockfile into the Nix derivative, at least in some cases. I actually can't remember why that has to be done, but presumably there is a good reason.
I think most multi-crate projects are only sorta accidentally including multiple versions of the same crate, which is why it's rather unfortunate that there isn't some abstraction between crate and module.
Making Cargo.toml is annoying enough especially if we're factoring things out from an existing crate to a separate one: obviously we'll need to specify all the deps that the code is already using, and also possibly remove some deps from the old crate's Cargo.toml, and it can take a good chunk of time just to put it together. I mean it's nothing that we can't deal with, but it does slow things down for no good reason.
> Making a cargo.toml is simple and frankly you don’t do it so often as to be a burden.
But is it so simple that it can be done often without having to go back to the docs and look up something, then making sure everything in the crate makes the build system happy? We get better at doing things the more often we do them, of course, so there's some inflection point where "often enough to remember how to do it" and "simple enough to be able to recall" cross.
With Go, making a new module is just creating a new directory, something programmers have done countless times, and requires no cooperation from the build tool. The only requirement is that files in directory have the same package declaration.
It is super easy to do. Cargo init --lib crates/something, done.. xD it is pretty much identical to go.
I've got other gripes with cargo.
- A go mod download equivalent would be nice so that we can actually reliably cache dependencies in ci.
- More sane linking tooling when statically compiling, eg. Openssl.
- Build.rs magic at compile time.
- No good way to publish a workspace of crates without actually having to publish everything.
- General workspace versioning unfriendliness. It is tricky to bump versions correctly in a workspace.
Probably some more. Most of these already have upstream issues but most have been stale for a while
Yeah, it maybe wasn't the greatest to throw blame at them, I really enjoy using rust and cargo. So I just want it to be the best it can be. Maybe I should look into what I can do to help them solve some of these issues =D
you have to include your source to be able to build your dependencies, this is not great for Docker builds, or just hashing in general. Cargo-chef tries to solve this, by only Including Cargo.toml and .lock, and creating main.rs and lib.rs files when it runs, it is a bit clunky though.
Basically you don't want a source change to invalidate your dependency fetching, which can occur if you use cargo fetch and just caching the target, especially with docker, as it doesn't have granular enough caching mechanisms.
with Go you have ONE go.mod file, at the module root. you may have 100 sub packages or sub sub sub packages, but still only ONE go.mod file. then any time you need to add a new import at ANY level, you just go "go mod tidy", and DONE. please dont try to compares apple and orange and say they are both apples.
There really isn't a huge difference, both have workspaces, and both allows you to add arbitrary go.mod or Cargo files where you want. The biggest difference is that go defines a module as a folder, where rust does it pr file. That said I don't know how that plays into how it is compiled.
Any of these subpackages, if they don't have a go.mod file, would require defining their dependencies top level as well, and when you go get the package you would need to pull everything, even if you only need a subpackage. Which usually isn't a problem, because golang is super fast to compile.
I get that they aren't identical, but tbh they are pretty close
Additionally Go requires search and replace all over the place, if a module ever changes it SCM location, and having internal build caches for the company, not even bother.
It's quite annoying in an ergonomic sense. Dependency management becomes harder, etc.
For instance, if you make more crates and they all use the same dependency, you have to manage it all in the multiple Cargo.toml files.
But overall, they're just two different systems with two different tradeoffs. I prefer writing Rust and I prefer the way I specify deps in Rust, but I prefer the ergonomics of packages in Go.
I do dislike how it forces entirety of code to be under single file.
I can't for example shove all of the definitions into single file then use it in other files without using sub-modules for no good reason, I much prefer "module = directory" approach to "module = file"
right but if you have two crates you have to list the common dependencies twice, and you have to either keep them in sync manually or put the crates into a workspace and then use workspace dependencies, so now you have to declare the dependent crate three times.
> So while Go encourages having reasonably sized packages, Rust unfortunately encourages huge crates.
I'm a bit confused by this as I'm used to seeing the opposite complaint, that Rust packages are too small. Maybe if there were examples it would help.
And while I do think we should explore ways to allow lower effort packages ("cargo script" is one component of this [0]), I see allowing submodules as a feature though maybe thats my Python/C++ background showing through. I do like that it allows both low overhead modules in your API (just a file) or allows you to separate a module out into multiple files for clearer organization (where cycles become important). While I don't know if there is a feasible design for this, but I do wish we could specify what modules are cyclic vs acyclic.
> 'm a bit confused by this as I'm used to seeing the opposite complaint, that Rust packages are too small.
By "packages" here you're probably referring to the public crates on crates.io, right?
Indeed, those public crates are not big in general, but that's not the point of the article. The article is rather talking about large, multi-crate projects, organized as a Cargo workspace. I agree it could have been clearer in the article.
What goes into another crate vs another module is an API architecture question; you can shake your fist at (overblown) compilation times but it doesn't make it bad practice to make stuff that's semantically a library, a library.
As an example of good practice, Go is pointed to, which is an ecosystem full of people splitting lots of stuff into separate libraries that make no sense to the end user as a separate library and in practice serve the role of submodules. This I would call bad practice, if not for the fact that Go disallows you from doing it any other way.
If Go had a central package index instead of just using GitHub, it would be a very confusing place, full of 'private submodules'.
What the author mentions but doesn’t seem to allocate a lot of significance to is that in Go, modules are a level of organization higher up the hierarchy from packages. Rust has no such a higher level organizing unit. It’s only unit for both sharing dependencies and compiling files is a crate. So crates have to take on additional duties, such as specifying build instructions and dependencies.
Been years since I’ve done much Rust. Does this mean bumping dependency versions thus becomes a larger chore in a project?
Rust has workspaces, which is a collection of packages (a Cargo.toml and related source files). Workspaces are still pretty underpowered, but they do have shared dependencies, compilation artifacts, version numbers, and CLI management. A lot of this is new (1 year old).
To me (a C++ daily driver and Rust hobbyist) this "DAG of strongly-connected" tiered graph system is sane and maps well to practice --- tightly couple your impl details however you want, then expose a clean interface at the crate level. It encourages you to split your code to separate files as you write _without immediately forcing you to untangle the dependency_, and then as "islands" of code organically form, which pieces belong to a separate crate becomes naturally evident.
Contrast this with C++ where the DAG relationship is enforced at much lower levels (e.g. Bazel cc_library, CMake target, which in normal cases correspond to a cc/h pair), shifting the friction (of having to deal with deps) much earlier in the dev cycle, which then discourages you from (incrementally) splitting a file at all. I've seen too many times this resulting in 5000+ LoC cc file.
> You might have noticed that when I'm talking about modules being organized as a tree, I say that it seems to be neatly organized. Because the dependencies are acyclic there, modules aren't actually organized, this tree structure merely provides an illusion, while underneath there is likely a dependency mess, and there can be hundreds of files in that mess, and Rust seemingly encourages that; otherwise why would we need modules to be organized in a tree in the first place.
Personally, this strikes me as bizarre. It is only an "illusion" if one expects a module tree structure to map one-to-one with dependencies. No one should expect this, as I will explain with three uncontroversial and widely known points:
A. Are there any languages where parent-child relationships between modules necessarily imply dependency semantics? I can't think of any, nor can I envision any sane language doing so. [Note 1]
B. In Rust, like with so many languages, when a module is nested in another module, that means the inner module is scoped inside the outer module. It is about namespacing, not dependencies.
C. Dependencies are created when the contents of something in a module refer to another module.
[Note 1]: How would such a language specify more than one dependency? A node in a tree can only have one parent, after all. This would rule out a one-to-one mapping.
Rust's trait coherence rules can make it impossible to split certain crates up. On the flip-side, cargo's workspace features make multi-crate projects easier to manage. I would love it if it were possible to loosen coherence within a workspace.
"private" trait impls (1) would be a much better solution. You probably never want the impl of a trait for a foreign type to be exposed to other crates, it's so unlikely that forbidding it in the name of coherence would probably work out. But there's apparently many gotchas and no one has clearly written a way around them (2).
(1) https://github.com/rust-lang/rfcs/issues/493 (although the comment section is kind of unhinged and goes off on distant tangents to modular typeclasses and first class modules)
The author desires a simple folder based module system and argues how it would improve compilation times. I do agree although it would be nice to know what flexibility would be lost so the argument could have some weight on the other side.
On a tangent I recall struggling with cargo features in the beginning. I found them to be hidden and mysterious. Even searching for “rust features” sucks.
You probably discovered this by now, but if you can find the Cargo.toml then the features are listed there. Or if you use 'cargo add' the features will be listed.
> So while Go encourages having reasonably sized packages, Rust unfortunately encourages huge crates.
Interesting since I think Rust very much encourages the opposite, which is why so many crates like hyper, regex, and others, are actually composed of many crates.
I don't really get the main thrust of the article. Go forces directories to map to packages. That sounds... awful. I guess it "encourages small packages" because you can't even organize them into folders? IDK, I feel like I'm missing something - particularly, I'm missing why big or small is good or bad, or why creating a crate is onerous.
Powerful features encourage bad practices. This is another "Rust empowers the developer, Go protects the developer from themselves" (and empowerment isn't always a good thing and protection-from-self a bad thing, e.g. goto, type systems)
First of all, what are other examples of "Rust empowers the developer, Go protects the developer from themselves"? As article mentions, Rust as a language is actually must stricter than Go, C++ and many others, and Rust community likes to brag that by restricting what we can do in safe Rust, we end up with better software. And I agree with that. But when it comes to dependency management, Rust turns out to be surprisingly lax.
Also, in this case, Go doesn't exactly protect the developer from anything. One could create a single package with tons of files in them, and it'll keep working. But the way Go package system is designed, it just subtly suggests to the developer that it's not the right thing to do: having a package (i.e. directory) with 200+ files in it just feels wrong by default. One trait of a well-designed system is that it's difficult to abuse.
I've never used Go, but do you really not need to create anything similar to a Cargo.toml file specifying dependencies to create a package? That doesn't seem like it could be true within Google since they would use blaze to compile their Go code, which mandates a new BUILD file for each package...
The other stuff here is mostly just a matter of convention. Every large Rust codebase I'm familiar with absolutely does avoid creating huge crates, so I think the author is just mistaken about what constitutes standard Rust practice.
There is no required cargo.toml equivalent for go. You put all dependencies at the top of your source code in an import statement. There are other things you can do if you want to lock a specific version of an API.
$ echo -e 'package main\n\nimport _ "golang.org/x/image/draw"' > main.go
$ go run main.go
main.go:3:8: no required module provides package golang.org/x/image/draw: go.mod file not found in current directory or any parent directory; see 'go help modules'
$ go build main.go
<same error>
go.mod seems pretty required.
With rust, you can also technically just call 'rustc' and never use 'cargo', but just like it's a pain to do that in go, it's a pain to do that in rust.
I get that cyclic dependencies can appear[0] to encourage bad design but that this automatically yields larger crates does not seem evident or proven here...
[0] this is probably a bad opinion from someone who's honestly never designed large systems, but it just seems to me that sometimes cycles are inevitable, you just have to be careful when they arise and minimize the times they happen.
1) he never says what bad practices are encouraged. A crate can have many source files, so what's wrong with big crates? It's better than npm with tons of stupid small dependencies.
2) what if 2 modules in go share a common dependency? It's not clear to me how that's handled in a file system hierarchy. But thats beside the point anyway.
Excellent Article !! Infact this article helped me make a choice. To learn Go or Rust. Clearly, unless the project is very lahttps://news.ycombinator.com/item?id=36233862rge or demands rewrite of entire stack, then GO is the choice
All this rust to learn Rust seems more like a gold rush to me.
Go seems to be the python equivalent of systems programming
There is one giant issue with Rust crates vs modules, and it's not "crates are harder to make."
The real problem is that you can't use crates to improve incrementalism when you have a type in one logical part of the module hierarchy implement a trait from another logical part. Due to orphan rules and coherence, it is not possible to implement a foreign trait for a foreign type.
And that said, just because the compilation unit is a crate doesn't mean it can't be compiled incrementally or with parallelism. That's a separate problem.
> The real problem is that you can't use crates to improve incrementalism when you have a type in one logical part of the module hierarchy implement a trait from another logical part. Due to orphan rules and coherence, it is not possible to implement a foreign trait for a foreign type.
Well you can implement a trait for a type from another logical part either in the crate of the trait or the crate of the type.
What is the use-case for implementing a foreign trait for a foreign type if you have both under control?
One of the difficulties of breaking up a large crate is the Orphan Rule. I got bitten by it once. I had some traits and when I broke up the crate suddenly some types but also the traits were in different crates each.
I wanted to separate the type, the trait and the implementation of the trait in different crates. I was forced to use newtypes but newtypes need a lot of boiler plate. It was painful.
In my opinion this is a larger barrier than having separate Cargo.toml files.
> I wanted to separate the type, the trait and the implementation of the trait in different crates.
I usually put the traits in a separate crate, but keep concrete types and implementations for them in the same crate.
What is the use-case for splitting a type and it's implementation into separate crates? (except for cases where the trait or the type are out of your control, then one indeed needs to use the new-type escape hatch)
I had three different goals, one for the traits, one for something else where the types fit best, and the third which I achieved with the implementations of the traits.
The problem was that the trait implementations of the types was something like not a core task of the types themselves.
Excellent Article !! Infact this article helped me make a choice. To learn Go or Rust. Clearly, unless the project is very large or demands rewrite of the entire stack, then GO is the choice
All this rust to learn Rust seems more like a gold rush to me.
Go seems to be the python equivalent of systems programming
It may be that what Rust allows here is not perfect, but it is not exactly bad. Certainly, compared to most languages, there is nothing abnormal about allowing many files in a tree of directories in a crate, gem, egg, rock, package, module, &c.
I was expecting something about how you can't move code into a new file without creating a new module, which means you'll need to modify calling code if you want to reorganize stuff, so you end up with massive source files.
You mean that if you split a file you define another module when you declare it? That is only visible as a module at the level of declaration not beyond.
Also note that modules are not per se compile units.
This article considers design problems introduced by humans, not the language. Cycles aren't a natural side effect of using Rust but rather a side effect of modularization.
The people cheering on Go here surprise me a bit. I don't hate Go, but I have spent way more time reading stack overflow posts trying to get a go project to be glued together, fiddling with environment variables, copy pasting code around then I ever have in rust.
Again not claiming superiority of one tool over another... But the user experience for rust has been a lot better for project organization for me then most languages. My biggest gripe is, the rust docs somehow make this incredibly difficult language easy to learn, but the sections around code organization are really really hard for beginners or anyone who hasn't sunk time into it and completely omit the 10 meter view of how to organize rust code.
I feel like the author really overstates how hard it is to make a new crate. Making a cargo.toml is simple and frankly you don’t do it so often as to be a burden.
Encouraging cyclic dependencies on the other hand “because it is easy” seems like a lazy cop out.
There are some efforts for lowering the overhead and would be interesting to explore how we could further do it.
First, we allow sharing `Cargo.toml` fields across the workspace since 1.64.0 [0]. In the upcoming 1.71.0, we extend this so that `cargo new` will automatically inherit fields when creating new `Cargo.toml` files [1].
From a different angle, we are also adding support for single-file packages though this is more meant for binaries [2]
[0] https://doc.rust-lang.org/cargo/reference/workspaces.html#th...
[1] https://github.com/rust-lang/cargo/pull/12069
[2] https://github.com/rust-lang/rfcs/pull/3424
Well, they seem to be cognizant of that, but they have a point: compared to simply creating a folder, it is QUITE a lot more effort. It's like the difference between zero and one.
That having been said, I think my bigger issue with this is that this actually gets to be annoying after you do it, not just while doing it. Go manages external dependencies on the module level, which is a level higher than packages. For Rust, it's crates. So a repo with many crates becomes pretty cumbersome. (And a bit moreso if you're trying to maintain a Nix package for said repo, but that's partly the fault of Nix. Only partly, because I actually think Rust's toolchain makes the job of Nix rather difficult...)
As a moderate Nix user/packager, I feel like the ideal model for ecosystem packaging in Nix is poetry2nix, where the existing lockfile hashes are able to be used directly to create pure Nix builds, with no need to generate code, pre-download anything, or have opaque "deps" blobs (like with Bazel).
As far as I can tell, all or most of these requirements are met by Cargo lockfiles as well, so what is the gap that makes Cargo/Rust difficult to do well in Nix?
The main issue I've come across is that when dealing with multi-crate projects, it is not uncommon to have two versions of the exact same version number of a Rust crate in two different dependencies. Due to a limitation in the machinery, this is painful.
The other thing that's weird is that you have to embed the lockfile into the Nix derivative, at least in some cases. I actually can't remember why that has to be done, but presumably there is a good reason.
I think most multi-crate projects are only sorta accidentally including multiple versions of the same crate, which is why it's rather unfortunate that there isn't some abstraction between crate and module.
Making Cargo.toml is annoying enough especially if we're factoring things out from an existing crate to a separate one: obviously we'll need to specify all the deps that the code is already using, and also possibly remove some deps from the old crate's Cargo.toml, and it can take a good chunk of time just to put it together. I mean it's nothing that we can't deal with, but it does slow things down for no good reason.
Copy over all the dependencies and then run something like this:
https://blog.benj.me/2022/04/27/cargo-machete/
Maybe something like that should be built into the Rust toolchain, rather than being a third-party tool?
Maybe one day. In the mean time, it is best for many people to invent many tools and then find out which work the best for the most people.
Agreed. Is there a community process for proposing changes and enhancements to the tooling?
There is, Cargo uses the RFC process for big changes just like rustc.
However, the Cargo team has been stretched a bit thin; back in April they added two new team members, and so things look like they're going in the right direction again: https://blog.rust-lang.org/inside-rust/2023/04/06/cargo-new-...
> Making a cargo.toml is simple and frankly you don’t do it so often as to be a burden.
But is it so simple that it can be done often without having to go back to the docs and look up something, then making sure everything in the crate makes the build system happy? We get better at doing things the more often we do them, of course, so there's some inflection point where "often enough to remember how to do it" and "simple enough to be able to recall" cross.
With Go, making a new module is just creating a new directory, something programmers have done countless times, and requires no cooperation from the build tool. The only requirement is that files in directory have the same package declaration.
It is super easy to do. Cargo init --lib crates/something, done.. xD it is pretty much identical to go.
I've got other gripes with cargo.
- A go mod download equivalent would be nice so that we can actually reliably cache dependencies in ci. - More sane linking tooling when statically compiling, eg. Openssl. - Build.rs magic at compile time. - No good way to publish a workspace of crates without actually having to publish everything. - General workspace versioning unfriendliness. It is tricky to bump versions correctly in a workspace.
Probably some more. Most of these already have upstream issues but most have been stale for a while
To be fair, it seems Cargo team has been understaffed for a long time. So developement has been kinda stalling. They are trying to fix it.
Yeah, it maybe wasn't the greatest to throw blame at them, I really enjoy using rust and cargo. So I just want it to be the best it can be. Maybe I should look into what I can do to help them solve some of these issues =D
> A go mod download equivalent would be nice so that we can actually reliably cache dependencies in ci.
What does go mod download do? cargo fetch/cargo vendor don't do the trick?
cargo fetch gets most of the way there. But you would need to build them as well to actually get good performance.
This is the upstream issue in question: https://github.com/rust-lang/cargo/issues/2644
Cargo fetch and/or cargo-chef kind of does these things, but not in a super intuitive and easy to use way.
you have to include your source to be able to build your dependencies, this is not great for Docker builds, or just hashing in general. Cargo-chef tries to solve this, by only Including Cargo.toml and .lock, and creating main.rs and lib.rs files when it runs, it is a bit clunky though.
Basically you don't want a source change to invalidate your dependency fetching, which can occur if you use cargo fetch and just caching the target, especially with docker, as it doesn't have granular enough caching mechanisms.
Caching ~/.cargo is great tho =D
> it is pretty much identical to go.
Its not.
with Go you have ONE go.mod file, at the module root. you may have 100 sub packages or sub sub sub packages, but still only ONE go.mod file. then any time you need to add a new import at ANY level, you just go "go mod tidy", and DONE. please dont try to compares apple and orange and say they are both apples.
There really isn't a huge difference, both have workspaces, and both allows you to add arbitrary go.mod or Cargo files where you want. The biggest difference is that go defines a module as a folder, where rust does it pr file. That said I don't know how that plays into how it is compiled.
Any of these subpackages, if they don't have a go.mod file, would require defining their dependencies top level as well, and when you go get the package you would need to pull everything, even if you only need a subpackage. Which usually isn't a problem, because golang is super fast to compile.
I get that they aren't identical, but tbh they are pretty close
Additionally Go requires search and replace all over the place, if a module ever changes it SCM location, and having internal build caches for the company, not even bother.
This is far from the only way to work with Go, and is one of the worst, as it leads to uncontrolled dependency sprawl.
Go has workspaces now. IMO they kind of suck to use, but they do exist. https://go.dev/doc/tutorial/workspaces
It's quite annoying in an ergonomic sense. Dependency management becomes harder, etc.
For instance, if you make more crates and they all use the same dependency, you have to manage it all in the multiple Cargo.toml files.
But overall, they're just two different systems with two different tradeoffs. I prefer writing Rust and I prefer the way I specify deps in Rust, but I prefer the ergonomics of packages in Go.
> I feel like the author really overstates how hard it is to make a new crate.
I read it more as the maintenance overhead when publishing. You need to bump version, check dependency versions etc.
I'm speaking from personal experience here. The more I split my crates into sub-crates, the longer the release/publish process takes.
There are some parts you can automate with e.g. cargo-release but I now think a lot harder before cutting a release to crates.io.
I also have the gut feeling this is happening throughout the ecosystem.
I have a lot more overlays in my Cargo.tomls now than I used to, two years ago.
I.e. there are PRs and fixes in the repo. that you need but the author hasn't cut a release.
This may be coincidence ofc but I can't help but think there may be a connection.
I do dislike how it forces entirety of code to be under single file.
I can't for example shove all of the definitions into single file then use it in other files without using sub-modules for no good reason, I much prefer "module = directory" approach to "module = file"
Why are you linking a document showing module need to be single file ?
right but if you have two crates you have to list the common dependencies twice, and you have to either keep them in sync manually or put the crates into a workspace and then use workspace dependencies, so now you have to declare the dependent crate three times.
> So while Go encourages having reasonably sized packages, Rust unfortunately encourages huge crates.
I'm a bit confused by this as I'm used to seeing the opposite complaint, that Rust packages are too small. Maybe if there were examples it would help.
And while I do think we should explore ways to allow lower effort packages ("cargo script" is one component of this [0]), I see allowing submodules as a feature though maybe thats my Python/C++ background showing through. I do like that it allows both low overhead modules in your API (just a file) or allows you to separate a module out into multiple files for clearer organization (where cycles become important). While I don't know if there is a feasible design for this, but I do wish we could specify what modules are cyclic vs acyclic.
[0] https://github.com/rust-lang/rfcs/pull/3424
> 'm a bit confused by this as I'm used to seeing the opposite complaint, that Rust packages are too small.
By "packages" here you're probably referring to the public crates on crates.io, right?
Indeed, those public crates are not big in general, but that's not the point of the article. The article is rather talking about large, multi-crate projects, organized as a Cargo workspace. I agree it could have been clearer in the article.
What goes into another crate vs another module is an API architecture question; you can shake your fist at (overblown) compilation times but it doesn't make it bad practice to make stuff that's semantically a library, a library.
As an example of good practice, Go is pointed to, which is an ecosystem full of people splitting lots of stuff into separate libraries that make no sense to the end user as a separate library and in practice serve the role of submodules. This I would call bad practice, if not for the fact that Go disallows you from doing it any other way.
If Go had a central package index instead of just using GitHub, it would be a very confusing place, full of 'private submodules'.
https://dave.cheney.net/2019/10/06/use-internal-packages-to-...
Indeed Go's "internal" packages serve the need to create packages that are not libraries
leftpad
What the author mentions but doesn’t seem to allocate a lot of significance to is that in Go, modules are a level of organization higher up the hierarchy from packages. Rust has no such a higher level organizing unit. It’s only unit for both sharing dependencies and compiling files is a crate. So crates have to take on additional duties, such as specifying build instructions and dependencies.
Been years since I’ve done much Rust. Does this mean bumping dependency versions thus becomes a larger chore in a project?
Rust has workspaces, which is a collection of packages (a Cargo.toml and related source files). Workspaces are still pretty underpowered, but they do have shared dependencies, compilation artifacts, version numbers, and CLI management. A lot of this is new (1 year old).
Cargo workspaces have existed for a number of years now - the feature was released in Cargo alongside Rust 1.12 in September 2016: https://blog.rust-lang.org/2016/09/29/Rust-1.12.html
The version numbers are from a little more than a year ago. Originally, it was pretty much only compilation artifacts and some CLI. https://github.com/rust-lang/cargo/pull/10563
To me (a C++ daily driver and Rust hobbyist) this "DAG of strongly-connected" tiered graph system is sane and maps well to practice --- tightly couple your impl details however you want, then expose a clean interface at the crate level. It encourages you to split your code to separate files as you write _without immediately forcing you to untangle the dependency_, and then as "islands" of code organically form, which pieces belong to a separate crate becomes naturally evident.
Contrast this with C++ where the DAG relationship is enforced at much lower levels (e.g. Bazel cc_library, CMake target, which in normal cases correspond to a cc/h pair), shifting the friction (of having to deal with deps) much earlier in the dev cycle, which then discourages you from (incrementally) splitting a file at all. I've seen too many times this resulting in 5000+ LoC cc file.
Modules are similar high level.
Naturally you can only enjoy them today on VC++, but the rest will follow.
> You might have noticed that when I'm talking about modules being organized as a tree, I say that it seems to be neatly organized. Because the dependencies are acyclic there, modules aren't actually organized, this tree structure merely provides an illusion, while underneath there is likely a dependency mess, and there can be hundreds of files in that mess, and Rust seemingly encourages that; otherwise why would we need modules to be organized in a tree in the first place.
Personally, this strikes me as bizarre. It is only an "illusion" if one expects a module tree structure to map one-to-one with dependencies. No one should expect this, as I will explain with three uncontroversial and widely known points:
A. Are there any languages where parent-child relationships between modules necessarily imply dependency semantics? I can't think of any, nor can I envision any sane language doing so. [Note 1]
B. In Rust, like with so many languages, when a module is nested in another module, that means the inner module is scoped inside the outer module. It is about namespacing, not dependencies.
C. Dependencies are created when the contents of something in a module refer to another module.
[Note 1]: How would such a language specify more than one dependency? A node in a tree can only have one parent, after all. This would rule out a one-to-one mapping.
Rust's trait coherence rules can make it impossible to split certain crates up. On the flip-side, cargo's workspace features make multi-crate projects easier to manage. I would love it if it were possible to loosen coherence within a workspace.
"private" trait impls (1) would be a much better solution. You probably never want the impl of a trait for a foreign type to be exposed to other crates, it's so unlikely that forbidding it in the name of coherence would probably work out. But there's apparently many gotchas and no one has clearly written a way around them (2).
(1) https://github.com/rust-lang/rfcs/issues/493 (although the comment section is kind of unhinged and goes off on distant tangents to modular typeclasses and first class modules)
(2) http://smallcultfollowing.com/babysteps/blog/2015/01/14/litt...
The author desires a simple folder based module system and argues how it would improve compilation times. I do agree although it would be nice to know what flexibility would be lost so the argument could have some weight on the other side.
On a tangent I recall struggling with cargo features in the beginning. I found them to be hidden and mysterious. Even searching for “rust features” sucks.
You probably discovered this by now, but if you can find the Cargo.toml then the features are listed there. Or if you use 'cargo add' the features will be listed.
Took me a bit too at first.
> So while Go encourages having reasonably sized packages, Rust unfortunately encourages huge crates.
Interesting since I think Rust very much encourages the opposite, which is why so many crates like hyper, regex, and others, are actually composed of many crates.
I don't really get the main thrust of the article. Go forces directories to map to packages. That sounds... awful. I guess it "encourages small packages" because you can't even organize them into folders? IDK, I feel like I'm missing something - particularly, I'm missing why big or small is good or bad, or why creating a crate is onerous.
I think the author is unaware of a) how to create crates easily b) that workspaces exist to organize crates
I wouldn't pick Go as an example of how to build a module system. Far from it. ML languages yes.
Anyways, adding a single line command per compile unit package is not a "cost."
Edit: P.S. Also the module logical system in Rust is way much more expressive.
Powerful features encourage bad practices. This is another "Rust empowers the developer, Go protects the developer from themselves" (and empowerment isn't always a good thing and protection-from-self a bad thing, e.g. goto, type systems)
First of all, what are other examples of "Rust empowers the developer, Go protects the developer from themselves"? As article mentions, Rust as a language is actually must stricter than Go, C++ and many others, and Rust community likes to brag that by restricting what we can do in safe Rust, we end up with better software. And I agree with that. But when it comes to dependency management, Rust turns out to be surprisingly lax.
Also, in this case, Go doesn't exactly protect the developer from anything. One could create a single package with tons of files in them, and it'll keep working. But the way Go package system is designed, it just subtly suggests to the developer that it's not the right thing to do: having a package (i.e. directory) with 200+ files in it just feels wrong by default. One trait of a well-designed system is that it's difficult to abuse.
I've never used Go, but do you really not need to create anything similar to a Cargo.toml file specifying dependencies to create a package? That doesn't seem like it could be true within Google since they would use blaze to compile their Go code, which mandates a new BUILD file for each package...
The other stuff here is mostly just a matter of convention. Every large Rust codebase I'm familiar with absolutely does avoid creating huge crates, so I think the author is just mistaken about what constitutes standard Rust practice.
sibling comment is wrong. this is the typical flow on a new package:
1. go mod init something # create go.mod
2. go mod tidy # create go.sum
3. have fun
That’s a new module in Go parlance. Making a new package just requires making a new directory.
There is no required cargo.toml equivalent for go. You put all dependencies at the top of your source code in an import statement. There are other things you can do if you want to lock a specific version of an API.
Okay, let me try that:
go.mod seems pretty required.With rust, you can also technically just call 'rustc' and never use 'cargo', but just like it's a pain to do that in go, it's a pain to do that in rust.
With Go at least, just prepend GO111MODULE=off and it works.
A bit esoteric, but I like that option, especially when working on an application and its dependencies in parallel at times.
I get that cyclic dependencies can appear[0] to encourage bad design but that this automatically yields larger crates does not seem evident or proven here...
[0] this is probably a bad opinion from someone who's honestly never designed large systems, but it just seems to me that sometimes cycles are inevitable, you just have to be careful when they arise and minimize the times they happen.
1) he never says what bad practices are encouraged. A crate can have many source files, so what's wrong with big crates? It's better than npm with tons of stupid small dependencies.
2) what if 2 modules in go share a common dependency? It's not clear to me how that's handled in a file system hierarchy. But thats beside the point anyway.
Excellent Article !! Infact this article helped me make a choice. To learn Go or Rust. Clearly, unless the project is very lahttps://news.ycombinator.com/item?id=36233862rge or demands rewrite of entire stack, then GO is the choice
All this rust to learn Rust seems more like a gold rush to me.
Go seems to be the python equivalent of systems programming
There is one giant issue with Rust crates vs modules, and it's not "crates are harder to make."
The real problem is that you can't use crates to improve incrementalism when you have a type in one logical part of the module hierarchy implement a trait from another logical part. Due to orphan rules and coherence, it is not possible to implement a foreign trait for a foreign type.
And that said, just because the compilation unit is a crate doesn't mean it can't be compiled incrementally or with parallelism. That's a separate problem.
> The real problem is that you can't use crates to improve incrementalism when you have a type in one logical part of the module hierarchy implement a trait from another logical part. Due to orphan rules and coherence, it is not possible to implement a foreign trait for a foreign type.
Well you can implement a trait for a type from another logical part either in the crate of the trait or the crate of the type.
What is the use-case for implementing a foreign trait for a foreign type if you have both under control?
One of the difficulties of breaking up a large crate is the Orphan Rule. I got bitten by it once. I had some traits and when I broke up the crate suddenly some types but also the traits were in different crates each.
I wanted to separate the type, the trait and the implementation of the trait in different crates. I was forced to use newtypes but newtypes need a lot of boiler plate. It was painful.
In my opinion this is a larger barrier than having separate Cargo.toml files.
> I wanted to separate the type, the trait and the implementation of the trait in different crates.
I usually put the traits in a separate crate, but keep concrete types and implementations for them in the same crate.
What is the use-case for splitting a type and it's implementation into separate crates? (except for cases where the trait or the type are out of your control, then one indeed needs to use the new-type escape hatch)
I had three different goals, one for the traits, one for something else where the types fit best, and the third which I achieved with the implementations of the traits.
The problem was that the trait implementations of the types was something like not a core task of the types themselves.
Excellent Article !! Infact this article helped me make a choice. To learn Go or Rust. Clearly, unless the project is very large or demands rewrite of the entire stack, then GO is the choice
All this rust to learn Rust seems more like a gold rush to me.
Go seems to be the python equivalent of systems programming
It may be that what Rust allows here is not perfect, but it is not exactly bad. Certainly, compared to most languages, there is nothing abnormal about allowing many files in a tree of directories in a crate, gem, egg, rock, package, module, &c.
I was expecting something about how you can't move code into a new file without creating a new module, which means you'll need to modify calling code if you want to reorganize stuff, so you end up with massive source files.
You mean that if you split a file you define another module when you declare it? That is only visible as a module at the level of declaration not beyond.
Also note that modules are not per se compile units.
Cf. See https://doc.rust-lang.org/rust-by-example/mod/split.html
Edit: typo.
You can fix this by doing `pub use module::*`.
wildcard import, definitely no drawbacks to those!
This article considers design problems introduced by humans, not the language. Cycles aren't a natural side effect of using Rust but rather a side effect of modularization.
Move your shared dependencies into another crate.
Rust 's flexibility exists for a reason, and it isn't as if the whole Go modules aren't without issues.
Not to mention the whole bonkers idea of having package names bound to Github repos and source only.
The people cheering on Go here surprise me a bit. I don't hate Go, but I have spent way more time reading stack overflow posts trying to get a go project to be glued together, fiddling with environment variables, copy pasting code around then I ever have in rust.
Again not claiming superiority of one tool over another... But the user experience for rust has been a lot better for project organization for me then most languages. My biggest gripe is, the rust docs somehow make this incredibly difficult language easy to learn, but the sections around code organization are really really hard for beginners or anyone who hasn't sunk time into it and completely omit the 10 meter view of how to organize rust code.
[dead]