> In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies
We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager.
For this to work at scale we'd need to overcome a couple of large problems though (in ascending order of complexity):
1. A better technical implementation of the model (working on it [1]).
2. A mindset shift to make people understand that "binary distribution" is not a goal, but a side-effect of a reasonable software addressing and caching model. Without this conceptual connection, everything is 10x harder (which is why e.g. Debian packaging is completely incomprehensible - their fundamental model is wrong).
3. A mindset shift to make people understand that their pet programming language is not actually a special snowflake. No matter what the size of your compilation units is, whether you call modules "modules", "classes" or "gorboodles", whether you allow odd features like mutually-recursive dependencies and build-time arbitrary code execution etc.: Your language fits into the same model as every other language. You don't have to NIH a package manager.
This last one is basically impossible at the current stage. Maybe somewhere down the line, if we manage to establish such a model successfully in a handful of languages and people see for themselves, but for now we have to just hold out.
And Guix has put that in an beautiful form. There are two things which make Guix special:
1. The package definitions are just a normal, battle-proven, very well defined general-purpose, functional-style supporting programming language (Scheme).
2. There is no conceptual difference between a package definition in the public Guix system, and a self-written package definition which a developers makes to build and test his own package, or to build and run a specific piece of software. The difference is equally small as between using an Emacs package, and configuring that package in ones .emacs configuration file.
I think 3 is probably true for language developers, but for users, language repos feel like they are needed because like none of the linux distros really ship everything you need, or even a reliable fraction without installing a bunch of dumb hacks to get it to work. It's much easier just to do "pip install" so that's where the demand is.
And sure, may be you're right that distro packaging is the "wrong model," again, that is the problem then for distros, users are stuck using ubuntu or whatever so they don't have the option to do the "right" thing, so they do use the mishmash of packaging/repo systems as just the cost of doing business.
Maybe our definition of "easy" doesn't agree, but for Ksplice, we had to build kernels using the exact same, often old versions of GCC and everything else (binutils, etc). We made schroots (this predated docker) and then we used the system package manager to install the requisite old version of packages. Wasn't trivial to get the whole system setup, but getting old versions of packages installed wasn't impossibly hard either.
Nix is not capable of becoming a first class dependency manager because it does not manage dependencies. It does not attempt any version or feature resolution. Besides, saying Nix could become a package manager is like saying that C could become a package manager.
Under this abuse of terminology, a pocket calculator is capable of becoming a package/dependency manager. Or any other technology which conceivably could be put to work toward the package management problem, but has not yet been adapted to that need. I concede that these statements are "true", but they fail the relevancy test of communication.
Like I said in another comment, the problem is curating the giant database that maps version numbers to commit refs. Once you pass this roadblock, the tools Nix gives you make dependency resolution a quite a bit simpler problem to solve.
The next step is then realizing you actually don't need the useless legacy version numbering scheme at all, and that you wasted that effort for nothing.
What we actually want is a content-addressable global namespace of source code. (Maybe it could be marketed as "blockchain for software" or something, lol.)
Version numbers are an attempt at content-addressing from the time when software came on floppy disks and CD-ROMs. It sorta made sense given the constraints of that technology.
Nowadays trusting git tags over git revision hashes makes no real sense. (Except that git tags are kinda shorter to type, but who cares.)
Pretend I asked the nice version of this, for the benefit of other readers?
I do think a Nix-like tool to be used for writing/replacing package managers for new programming languages would benefit from some dependency resolving functionality, instead of just saying 'shove it in a monorepo or pin via VCS refs instead of semver'. Is that not necessary?
At least for (unlocked) flake sources, you could probably use a branch naming convention instead of a separate database, although that requires your repo to be set up a specific way.
(I actually think Nixpkgs could and should go down this road, with a couple of caveats, for immense benefit. But I'm curious about what tazjin thinks here.)
2) In Debian a binary package has almost always just been a cache, are we thinking about Debian packaging in completely different ways? Are you perhaps talking about the aspect of doing an "build world"[0] that bsd, Gentoo and now Nix are better at?
[0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in class by far. It is also completely different from the notion of compiling a program. I wonder what you consider the goal of a package system.
> are we thinking about Debian packaging in completely different ways?
Quite likely! The whole concept of separately building a source/binary package, and then uploading/"deploying" that binary package, already violates the notion of being "just a cache" for me. There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix (i.e. guarantees about the artifact being repeatable, and being addressable by its inputs rather than a human-curated tag or version number).
> Something that even Nix struggles with
Nix only struggles with it to the degree that some of the higher-level abstractions in the existing package set (which are built on top of the fundamental model, not part of it) can be confusing/underdocumented, but conceptually this is a simple thing to do in Nix. Even practically at this point it is usually simple - unless you're dealing with Haskell or Javascript, of course.
> I wonder what you consider the goal of a package system
I want it to let me describe a desired state, and then make it so. That state is best represented as a graph, like a Merkle tree, of the instructions for the individual steps and a way to address them. An individual step is a transformation (i.e. some program) executed over some sources, yielding some result (likely in the filesystem). I want any distribution of binaries to be a result of using this addressing scheme, and looking up/copying an already built equivalent artifact for that thing. I want a text file containing the string "Hello HN" to be represented by the same abstraction that represents a specific .so file, the `emacs` package, the configuration of my entire system, the configuration of a cluster of systems and so on. I want this system to be programmable so I can work around the shortcomings that its original designers missed.
Nix (and Guix) do large parts of this already, and are conceptually (though not yet technically) capable of doing all of it. Technically, so is something like Bazel - but its complexity and maintenance requirements make that prohibitively expensive (basically only Google can use Bazel like that, and even they have delineated areas where they "give up" on wrapping things in Bazel).
> There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix
It's been there for 15 or 20 years. Also Debian has been doing reproducible builds before Nix.
Even in the best case where Debian satisfies all the reproducibility goals it sets for itself (i.e. the ~96% at https://tests.reproducible-builds.org/debian/reproducible.ht... becomes 100%), each Debian package is still only a reproducible function of the current versions of its build dependencies at the time it was built. Upgrading those dependencies can change the build output, and this is still normal and expected in Debian; it’s the entire point of a binNMU.
In order to reproduce a given Debian package from scratch, one would need to track down long chains of historical versions of packages from sid. Due to circular dependencies, these chains probably snake back to the very first versions of Debian before the availability of snapshot.debian.org. It’s possible that human judgment could be used to break some chains at an earlier point by substituting different versions, but this goes outside the guarantees provided by Debian’s reproducibility.
Nix guarantees that every package is built with the specified versions of its dependencies, and so on recursively. Some more rebuilding is necessary to maintain this guarantee, but in exchange, it enables stronger notions of reproducibility that Debian can’t.
Right. Nix bootstraps from a fixed collection of seed binaries, and everything in any given commit of nixpkgs is derived from the seed binaries and versions of software that exist in that commit of nixpkgs. It doesn’t solve the “trusting trust” problem with those seed binaries, but it does solve the “arbitrarily long chains” problem.
(The work at https://www.gnu.org/software/mes/ aims to minimize the needed binary seed to a human-auditable size, and should be applicable to Nix and Guix.)
fewer than 15 years ago. And it's not much like what the parent poster has in mind. It modifies the global state of your system, it requires multiple steps, it produces a different artifact than you might get in the repos at any given time, it leaves things in your CWD on your filesystem owned by your current user, whatever.
(Some of that may also apply to `apt-get source --build`, idk.)
With Nix or Guix, when a cached binary artifact for a given package is missing, you may not even notice, the fallback to building from source is so transparent.
I encourage you to actually try Nix or Guix, including writing packages for them, and including adding a patch or overriding a build flags in an existing package, to see how the model is really different.
> Also Debian has been doing reproducible builds before Nix.
What Debian does for reproducible builds and how Nix embodies reproducibility are qualitatively different. If you understood both, you'd understand why this is a silly point to raise.
(Debian's work on reproducible builds is great. Every distro benefits from most of it, and everyone in the Nix community is grateful for it.)
With Nix or Guix, when a cached binary artifact for a given package is missing, you may not even notice, the fallback to building from source is so transparent.
I consider this to be one of the main drawbacks of these systems -- it assumes that every machine is build-capable. I do not want compilers installed on my production servers (nor would I want my RPi to attempt to rebuild half the world because the package server was down). I'm not sure how Nix handles this, but I'm pretty sure that's not possible with Guix.
It doesn't, fortunately. You can tell Nix to rely exclusively on the binary cache, or configure it to have 0 build threads locally and use remote builders.
For a server deployment, many popular choices don't have the server build anything, and in some cases it may not even have any users who are allowed to invoke `nix-*` or anything.
To be fair, the whole concept of manually installing individual packages on production servers (especially when they run NixOS!) is a complete anti-pattern, so that situation wouldn't even occur during normal operation.
Debian packages can have pre/postinstall scripts that mutate the overall state of the system in arbitrary ways. This is both (a) necessary to make some thing work and (b) ruins the theoretical model.
Not necessary, but certainly it's easier to implement and get stuff done. It starts to break down and show its cracks the larger and larger it gets, though.
The difference between almost all Linux or BSD packaging systems and nixpkgs is like the difference between dynamic and compile-time-type-checked programming languages.
You can have a much more conventional package manager without pre- or post-install hooks. One ex-Debian developer created such a package manager as a research project, presumably to prove a point to other Debian people with the same head-in-the-sand attitude about the state of Debian's package tooling.
It doesn't require the same kind of patching as Nix does (although sometimes patching does help avoid requiring those scripts).
I'm in a team that works on a pet prog lang for distributed systems, and we did some research of using an existing package managing systems. We've settled on NPM for now, but god I wish there would be a better generic package manager out there.
Not a generic package manager, but it's probably worth calling out asdf as the generic version manager[0] (maybe you're already aware of it, but it's a generic replacement for nvm, rvm, virtualenv, *vm, which supports any language based on plugins.)
Again, maybe you're already aware of it, but I think it's a nice example of genericising a concern common to many languages which sounds similar to what you're asking for (albeit unfortunately in a slightly different space).
At a minimum it's useful having the `.tool-versions` dotfile in a project directory which lists the versions of each language, db, and package manager version (our Rails project lists versions for ruby, bundler, postgres, node, yarn, redis). Even if all devs don't use asdf, it's a useful reference point.
Check out Denxi. Might be the closest thing to what you're looking for today, and is informed by the state of the art in package management without totally imposing strict discipline on all packages.
If the community just focused on creating a great package manager instead of an Everything Monster, and “meeting developers where they are”, then Nix might do something great.
What's a good source, for someone who has never used nix, to read about its conceptual model and how it is different from contemporary dependency managers?
There really isn't one that I know of. There's the original thesis[0], but it's from a different time and it focuses only on the package management (~ for a distribution) aspect.
Everything else is mostly written to teach people how to use Nix, and the more recent the thing is the more it will focus on surface-level features of the C++ implementation of Nix.
We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager.
For this to work at scale we'd need to overcome a couple of large problems though (in ascending order of complexity):
1. A better technical implementation of the model (working on it [1]).
2. A mindset shift to make people understand that "binary distribution" is not a goal, but a side-effect of a reasonable software addressing and caching model. Without this conceptual connection, everything is 10x harder (which is why e.g. Debian packaging is completely incomprehensible - their fundamental model is wrong).
3. A mindset shift to make people understand that their pet programming language is not actually a special snowflake. No matter what the size of your compilation units is, whether you call modules "modules", "classes" or "gorboodles", whether you allow odd features like mutually-recursive dependencies and build-time arbitrary code execution etc.: Your language fits into the same model as every other language. You don't have to NIH a package manager.
This last one is basically impossible at the current stage. Maybe somewhere down the line, if we manage to establish such a model successfully in a handful of languages and people see for themselves, but for now we have to just hold out.
[0]: https://code.tvl.fyi/about/nix/buildGo
[1]: https://cs.tvl.fyi/depot/-/tree/tvix/