> The nice thing about Python packaging is that it’s standardized
But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library.
Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.
> if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.
> I don’t understand why Gentoo needs to rebuild every single release of uv-build.
Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.
> you can do a normal version bound in your Python build backend metadata
Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter".
distros patch the build process when they package stuff for the distro all the time. why would they "have to" do anything? i'm not super familiar with gentoo but it seems to me like any other 90s era distro where the distro mediates the relationship between the user and the upstream developer for all packages, by applying patches and potentially doing whatever build they want to.
Gentoo builds every package from source on every user's machine, and when each user does an update, every package that changed gets rebuilt from source again.
So every Gentoo user that has a Python project that uses uv-build ends up having to rebuild uv-build, and then every Python project they have that uses it, every time uv-build pushes an update.
Other distros don't necessarily have quite the same problem because, if they include packages of Python projects, they don't normally track the build backend as a dependency. So if uv-build pushes an update, the distro would push an updated package for uv-build, but they wouldn't look at every Python project they package to see if it used uv-build as a build backend and rebuild it if it did. It's the "rebuild from source every time" aspect of Gentoo that forces that connection.
i guess gentoo is rolling-release, vs something like debian where every package is frozen in time (+ backports), and debian does seem to largely solve the coordination problem by making everything on the system use a fixed version of some package; with rare situations allowing >1 version of the package at a time.
my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.
for example, they seem to be fine doing a quick sed on pyproject.toml (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/pa...) or patching stuff for compat (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/as...)
This is what i mean when it seems typical for a distro to monkey around with packages and/or their build system. These were just the first two packages i clicked on in dev-python. maybe others are "pure upstream", but there's definitely the capability here to just... not have the problem?
> i guess gentoo is rolling-release
That's my understanding, yes.
> my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.
Thing is, that apparently won't work. From [0], with some more discussion starting at [1]
> > I don’t understand why Gentoo needs to rebuild every single release of uv-build.
> Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.
[0] <https://news.ycombinator.com/item?id=50077466>
[1] <https://news.ycombinator.com/item?id=50077553>
Per my response, that isn’t how Python packaging works. There is no “use the latest version” mechanism in standard Python packaging; leaving a version unconstrained means that the installer, meaning Gentoo in this case, gets to decide how to constrain it.
> ...leaving a version unconstrained means that the installer, meaning Gentoo in this case, gets to decide how to constrain it.
So, would it be correct to decide that package with an unconstrained version gets built with 0.1.0? And when that fails to build, what's the distro packager to do? Bang on the package maintainer until they specify the real constraint? Yeah, good luck with that.
Doesn't it make way more sense to assume that "unconstrained uv-build version" means "they only tested this with the very latest version of uv-build" and proceed accordingly? This will fail sometimes, but I figure it'll fail far less often then assuming that you get to use 0.1.0 if no version is specified.
I'm being a little extreme with my example here, but only to illustrate the larger point I'm making. Breaking changes do happen, and without specified version constraints one has to assume that "very latest available" is the correct one.
Well, that’s why you should use a semver range, as is the default, instead of leaving it unconstrained. That’s the actual solution that solves the problems for all parties here; I’m just pointing out that “unconstrained = latest” isn’t a correct summary of how Python packaging works.
In that case, wouldn't it be better for uv to reject unconstrained? Gentoo (or any packager, such as homebrew / nix) is too far downstream for that. Presumably allowing unconstrained seemed to make sense in some circumstances at some point?
No, because it’s a standard part of Python packaging, and it’s not the problem here.
(There are all kinds of legitimate reasons to use an unconstrained dependency, including locking it separately and also trying to help other downstream Linux distros with their own bespoke packaging problems.)
you literally just said
You should probably pick one of these two mutually exclusive options and stick with it.
> Gentoo can't second guess you
gentoo appears to support exactly 7 versions of uv-build, a list they update by hand: https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/uv...
git changelog of uv-build in the gentoo package manager: https://gitweb.gentoo.org/repo/gentoo.git/log/dev-python/uv-...
gentoo also wrap python packages in their own package manager, packages in this tree appear to come from their upstream version control system eg Github, not PyPI.
just because someone on hn said something, doesn't make it true.
> they don't normally track the build backend as a dependency... It's the "rebuild from source every time" aspect of Gentoo that forces that connection
I don't follow. This feels like Gentoo is mishandling build-tool dependencies.
Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?
Strictly speaking, they should, right? Differences in GCC could produce differences in anything it compiles.
IIRC, rebuilds to GCC outputs are/were only recommended when the compiler "profile" changes. So basically whenever the ABI could no longer be guaranteed.
No, and not with clang or llvm or rust. When we get a new Golang tho, that does rebuild stuff built with Go. And big perl updates cause a rebuild for perl packages.
> I don't follow.
I don't either. I used to run Gentoo for quite a few years in the past and build dependency updates would not trigger rebuilds of build-dependent packages.
Rebuilding the entire system with a new GCC was recommended by the documentation, but there was nothing in Portage forcing it. Neither would you rebuild the system if, e.g. make got an update.
> This feels like Gentoo is mishandling build-tool dependencies.
I would say it's doing what it believes is necessary for the Python ecosystem. The build backend is an explicit part of the specification of a Python project.
> Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?
No, because C/C++ libraries don't have an explicit specification of the build tool to be used in their project, the way a Python project does.
> The build backend is an explicit part of the specification of a Python project
Nothing in the Python spec forces Gentoo to invalidate the downstream package every time build-system dependencies update - that's a behaviour they've decided is correct on their own.
> C/C++ libraries don't have an explicit specification of the build tool to be used in their project
They don't have a standard way to specify such, but they have an implicit dependency on the installed compiler toolchain, and it would be just as valid to interpret "no way to specify a build-system dependency" as "always rebuild" when the build tool updates
> Nothing in the Python spec forces Gentoo to invalidate the downstream package every time build-system dependencies update
Nothing in the Python spec guarantees that the resulting build artifact will be identical if the build system dependencies update but nothing else in the project does. So what's a "build everything from source" distro to do? If it's possible that updating the build dependency will change the build artifact, doesn't it have to rebuild?
> Nothing in the Python spec guarantees that the resulting build artifact will be identical if the build system dependencies update
With this framing, exactly the same thing can be said about libstdc++, for example, so must we also force a full system recompile on every GCC/clang version bump?
Seems like a design flaw in gentoo that previously hadn't been an issue because build tools weren't being updated nearly as often as uv-build is.
I linked to some of the relevant standards below.
At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way.
(And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.)
> it seems extremely unlikely that they actually need to build every single patch version of uv.
If every new patch version is still within the semver range specified in a project that uses uv-build, then yes, every Gentoo user will have to rebuild that new patch version of uv-build in order to rebuild that project. And since the build backend is a dependency of the project, pushing a new patch version will trigger a rebuild of the project.
> Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.
Sure it does. TFA claims
> all Gentoo users need to build it from source, and given that it’s released quite frequently (12 releases in the 30 days prior to writing this post), that’s a lot of building. Like, for every 100 ms spent running uv-build to install one release of your package, Gentoo users will spend many hours building new uv-build versions over and over again.
And that is false, exactly because of that standard. If you want reproducible builds then you can stick with a previously built version of uv-build for a long time. If you want working builds on the fly using your own processes, then the standard empowers you to substitute something else. If you want to use the Python packaging ecosystem in the normal way, then it doesn't even matter what your distro is.
uv-build is for pure Python projects, which entails that the "build" process is trivial. It doesn't even bytecode-compile the .pyc files generally; wheels aren't expected to contain them (your installer may produce them at install time, or you can let Python produce them on demand). Binary reproducibility for pure-Python wheels is just not, from anything I've seen, a high priority for Python packaging.
Linux distros have these ideas about packaging that are just not suitable for Python and not in tune with what the Python ecosystem is doing. The vast majority of the Python code you install should not be integrated into your "system". (Python devs rarely find themselves wanting to "bootstrap" everything from source, either.) Gentoo's issue here appears to be their own policy, which is a problem of their own making. It's unrealistic for them to expect random Python devs to care about this.
> As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.
If you mean because of Gentoo policy, that's their fault. I can guarantee you that approximately nobody using uv-build is concerned with whether their package makes it into Portage. Python build systems aren't designed to produce packages native to Portage (or any other Linux packaging system). Probably the overwhelming majority of these users aren't expecting any Linux distro repositories to host their package in any form, and probably most of them couldn't even tell you what Gentoo is.
If you mean technically, then no; once you have the source code and are using ebuild, nothing compels you to choose the backend specified in pyproject.toml. If everything is up to standard then it's trivial to switch it over to, say, Flit. Most projects following a standard src layout will need no tool-specific config, and there's quite little of it possible regardless (and all well documented). The author even highlights:
> It’s a backend and the whole point of PEP 517 is that you can use any backend with any tool. You can use flit-core, you can use hatchling, and either of them will work just fine with uv (or any other frontend).
The edits do take time but it's a lot better than the proposed bogeyman of waiting for a new build of uv-build from Rust source.