> OP handles the small change problem by hashing IR instead of source text
OP here---I think you got confused somewhere, because this isn't true! The hashes we take are of source code, they're just stored inside of the first IR. Source code is just pretty convenient to hash. If the change does turn out to be something trivial, we'll only invalidate the first-order dependencies of the source hash, which is never usually a big deal. In a particularly bad case, maybe we re-analyze, re-codegen, and re-link many different instances of a generic function, but even that wouldn't usually take long at all.
Also, it's actually important that we do notice these kinds of "trivial" changes in some parts of the pipeline---see Andrew's comment [0] on the Lobsters discussion for this post.
Oh, I forgot to respond to the "corrupted binary" thing in my other reply, sorry.
Right now, yep, corrupting the binary that the compiler reads would crash the compiler. In future, we want to detect the corruption and force a clean build. Note however that the Zig compiler is writing the binary to its internal cache directory (typically `.zig-cache/`), and the build system then copies the final artifact to your output ("prefix") directory (`zig-out/` by default), so it doesn't matter if the user messes with the final binary in `zig-out/bin/my_program`, because that's just a copy. Lastly, this is not implemented yet, but we will definitely make sure that Ctrl+C leave the cache in a clean state. I'm pretty sure our incremental compilation system has a nice property that you can just cancel an update partway through and continue it later without too much effort. See also Zig's IO interface [0] for information about cancelation. The compiler should already support graceful-ish cancelation internally (I won't claim to have verified this, because we never actually do cancel compilation right now, but I'm not aware of any glaring issues!). I don't think it'd be a crazy amount of work to improve that so that it also leaves incremental compilation in a valid state.
Zig uses a "single compilation unit" compilation model, so even if we did what you're suggesting, pretty much everything in the post would still be relevant, especially the whole-file stuff and the semantic analysis stuff. Note that there's not really anything special about putting things in separate source files in Zig---files are just a unit of code organization, like namespaces. They don't help the compiler at all (aside from in that very first part of compilation, but that's fast regardless).
The only thing I discussed that we could potentially avoid using your suggested strategy is incremental linking, but AFAICT that would only be easily avoidable if we made a separate shared object for each individual function. I guess we could group them arbitrarily and then re-run codegen for everything in one shared library when the other parts change, but that frankly doesn't sound much less awkward than incremental linking!
But putting all that aside: the only thing this approach would really achieve is offloading the linking work from the static linker to the dynamic linker (aka runtime linker). So if it did make compilation faster, I'd expect all of the saved time to just become runtime execution time---which is kind of worse, because the average number of times you run a compiled program is probably >1!
Of course, there's also the obvious point that dynamic linking only works in cases where you can run dynamic executables. For instance, that approach wouldn't work when doing operating system development, while our approach should work completely fine for that use case.
It works through the fact that I specified "runtime" function ;)
A bit after that quote I have a note about `inline` functions in Zig, where I mention that they perform semantic inlining, which means dependencies triggered by the function actually get associated with the call site. Well, `comptime` function calls work just the same way---in fact, to the compiler, `comptime` calls are almost exactly identical to `inline` calls. So when we encounter a comptime function call, we start analyzing the ZIR for that function's body, but we don't switch our analysis unit, so comptime stuff doesn't really complicate the dependency graph at all (aside from the fact that it means you can depend on any number of source code hashes, instead of everything depending on exactly one).
With all that being said, there actually is a (completely unrelated) way in Zig to can depend on the body of a runtime function (hence why the quote includes "at least in the simplified view I'm presenting here"). It's to do with "inferred error sets" (IESes for short). If a function's return type is written `!T`, that means it can return an error, but we're asking the compiler to figure out exactly which errors are possible. So if at some point we need to know that set of errors (e.g. because the user has done some reflection to try and access the list of errors), that's where we get a dependency on a runtime function body, because we need to analyze the function body to learn about all the places it might return an error.
It only works with our self-hosted code generation backends (the main one being for x86_64), which right now don't have any optimisation passes. It's planned that they will in future, but that's a long-term goal. Also, any optimisations which propagate information between functions (the most obvious and important one is inlining) are more-or-less incompatible with incremental compilation (I touch on this in the post IIRC), so once this does work it'll probably still be limited to a subset of optimisations.
TL;DR: only debug builds for now, could extend to release builds once we have our own optimisation passes one day, but some optimisations will still be inapplicable.
Zig team member here---obviously I can't say for sure yet, but I'm pretty confident the number will be basically identical. In the Zig compiler, incremental updates (rebuilds) have a small amount of overhead which is roughly proportional to the total size of the codebase (rather than just the amount of code which was changed). This comes from a) detecting which source files changed, and b) traversing a graph to figure out which declarations are referenced (necessary due to Zig's "lazy analysis" feature). But performance analysis reveals that for small updates, this overhead actually dominates the update time, by a lot. Of the 35ms, I would guess that under 5ms are actually spent rebuilding the function(s) that changed. Of that 5ms, code generation---the only thing which would really be different on AArch64---is an even smaller slice of the pie (it often doesn't even impact the overall time, since it runs in parallel with other parts of the pipeline, and those other parts are usually the bottleneck there). So even if the AArch64 backend was significantly slower (which, right now, is the opposite of what we expect---instruction selection and encoding for x86_64 is unusually complicated!), I wouldn't expect the number to change from 35ms.
Worth noting that while it's absolutely unfortunate that we're hitting this Forgejo design flaw, AGit honestly is just a better workflow, and I (Zig core team member here, full push access to the upstream repo) have started using it for pretty much all of my PRs. There are a couple of small things I'd like to see improved about Forgejo's implementation of AGit, but IMO it fundamentally makes so much more sense than the two-step push-then-PR workflow (especially if you're an external contributor who would also need a first step of "fork the repo").
`u3` would be base 8, i.e. octal---I think you meant to use `[400]u6`?
Aside from that: I'm not familiar with how standard base64 deals with endianness, so I'm not sure if it would match that, but this `@bitCast` would certainly give you a base64 encoding. But it would probably emit pretty terrible code to do that---our lowering of `@bitCast` isn't really optimized for moving around huge amounts of data in one operation! (But maybe LLVM would surprise me.)
Uh, no? My writing style just happens to include a lot of em-dashes, as is very common. And it's not like I'm pasting a weird Unicode codepoint all over the place, that's just (rightly) how my Markdown gets rendered...
OP here---I think you got confused somewhere, because this isn't true! The hashes we take are of source code, they're just stored inside of the first IR. Source code is just pretty convenient to hash. If the change does turn out to be something trivial, we'll only invalidate the first-order dependencies of the source hash, which is never usually a big deal. In a particularly bad case, maybe we re-analyze, re-codegen, and re-link many different instances of a generic function, but even that wouldn't usually take long at all.
Also, it's actually important that we do notice these kinds of "trivial" changes in some parts of the pipeline---see Andrew's comment [0] on the Lobsters discussion for this post.
[0]: https://lobste.rs/s/rmzzdb/inside_zig_s_incremental_compilat...