I was luck to know Josh Nimoy who is responsible for a lot of this in the movie, who has since sadly passed away. Josh took great pride in the fact that he was able to put Emacs and a bunch of Unix commands in a major Hollywood blockbuster.ed
FYI, (and sorry for the intrusion since you were a friend and I only learned of this person through this HN post) but it appears that you have used their deadname (which is an unfortunate term in this case).
Thank you for identifying them though, thanks to you I learned of a badass, and I regret the loss.
What stage is the "just make the compiler define the undefined" stage?
Unaligned access? Packed structs. Compiler will magically generate the correct code, as if it had always known how to do it right all along! Because it has, in fact, always known how to do it right. It just didn't.
Strict aliasing? Union type punning. Literally documented to work in any compiler that matters, despite the holy C standard never saying so. Alternatively, just disable it straight up: -fno-strict-aliasing. Enjoy reinterpreting memory as you see fit. You might hit some sharp edges here and there but they sure as hell aren't gonna be coming from the compiler.
Overflow? Just make it defined: -fwrapv. Replace +, -, * with __builtin_*_overflow while you're at it, and you even get explicit error checking for free. Nice functional interface. Generates efficient code too.
The "acceptance" stage is really "nobody sane actually cares about the C standard". The standard is garbage, only the compilers matter. And it turns out that compilers have plenty of extremely useful functions that let you side step most if not all of this. People just don't use this because they want to write "portable" "standard" C. The real acceptance is to break out of that mindset.
Somehow I built an entire lisp interpreter in freestanding C that actually managed to pass UBSan just by following the above logic. I was actually surprised at first: I expected it to crash and burn, but it didn't. So if I can do it, then anyone can do it too.
A lot of the Central UB can not be defined, because they rely on detection. In order to have a well defined behaviour (by the standard or the compiler) the implementation needs to first detect that the behaviour is triggered, this is often very tricky or expensive. Its easy to define that a program should halt, if it writes outside an array, but detecting if it does can be both slow and hard to implement. There are implementations that do, but they are rarely used outside of debugging.
A better way to think about UB is as a contract between developer and implementation, so that the implementations can more easily reason about the code. How would you optimize:
(x * 2) / 2
An optimizer can optimize this out for a signed integer, because it doesn't have to consider overflow, but with a unsigned integer it can not. UB is a big reason why C is the most power efficient high level language.
I don't even use * for multiplication anymore, I use __builtin_mul_overflow and then check the result. Anyone who doesn't is gonna hit the overflow case one day, and they'll be lucky if their program isn't exploited because of it. I've been making an effort to use all the overflow checking builtins by default in most if not all cases. I've also been making Claude audit every single bare arithmetic operation in my projects. He's caught quite a few security issues already, and overflow checking dealt with them all.
This particular contract between developer and implementation is totally worthless and doing more harm than good. It encompasses regular everyday normal things like multiplication and addition. All things that our brains literally rely on in order to reason about the code. Can't even add numbers without the compiler screwing it up.
Programmers need to deal with overflow at all times. Can't calculate an offset without dealing with overflow. Can't calculate a size without dealing with overflow. It's simply everywhere in systems programming, which is what C was designed to do. The consequence of ignoring this is usually that your program gets mercilessly exploited.
All this for some efficiency gains. The cost/benefit analysis is way off here. Things should be correct, first and foremost. Then the compiler should give us the necessary sharp tools to make it fast, if needed. It shouldn't be making it fast at the cost of turning the entire language into a memetic vulnerability machine.
The thing with (x * 2) / 2 is that for all practical purposes you might even have written something else, so the expression cannot be replaced by x directly.
What happens is that after a few common expression eliminations, peephole optmisations, code inlining, and possibly other optimisation passes, the remaining AST will be (x * 2) / 2, and then the magic happens.
That makes sense... I agree, and I don't really have an elegant answer to that at the moment.
I simply accepted the fact that the magic might stop happening at (x * 2) / 2. This acceptance bought me certainty about overflow behavior. I think it was a good tradeoff.
No. I like C. I've learned about a dozen languages by now. I always end up coming back to C. I've just accepted it.
There is no reason whatsoever that C can't be improved. Compiler attributes and builtins are already doing quite a lot of heavy lifting. Recent addition: counted_by, an attribute that allows compilers to properly track the size of memory referenced by pointers. All C programmers should be making liberal use of this stuff.
Packed structs are dangerous. You can do unaligned accesses through a packed type, but once you take the address of your misaligned int field, then you are back into UB territory. Very annoying in C++ when you try to pass the a misaligned field through what happens to be generic code that takes a const reference, as it will trigger a compiler warning. Unary operator+ is your friend.
> but once you take the address of your misaligned int field
Gotta work with the structure directly by taking the address of the packed structure itself.
struct uu64 {
u64 value;
} __attribute__((packed));
struct uu64 unaligned;
struct uu64 *address = &unaligned;
address->value; // this works
u64 *broken = &address->value; // this doesn't
Taking the address of the field inside the structure essentially casts away the alignment information that was explicitly added to stop the compiler from screwing things up. So it should not be done.
Mercifully, both gcc and clang emit address-of-packed-member warnings if it's done. So the packed structures are effectively turning silently broken nonsense code into sensible warnings. Major win.
> People just don't use this because they want to write "portable" "standard" C
Something that bothers me is the Venn diagram of people that think abstraction is slow and error prone and people that only write portable C.
How many C implementations do you actually need to compile against? I don't think I've seen more than 3 outside Unix software from the 90s. Using non portable extensions is in fact totally doable for your application and you should probably do it, and just duplicate/triplicate code where you have to. It's not that hard to write and not hard to read.
That's what I mean, I've seen enough autoconf "checking for <feature that totally works in every compiler you care about>" noise to know it's mostly pointless in this day in age.
> What stage is the "just make the compiler define the undefined" stage?
It can be left as implementation defined, which means that the compiler can't simply do arbitrary things, it needs to document what it would do.
Take, for example, signed-integer overflow: currently a compiler can simply refuse to emit the code in one spot while emitting it in another spot in the same compilation unit! Making it IB means that the compiler vendor will be forced to define what happens when a signed-integer overflows, rather than just saying, as they do now, "you cannot do that, and if you do we can ignore it, correct it, replace it or simply travel back in time and corrupt your program".
> Somehow I built an entire lisp interpreter in freestanding C that actually managed to pass UBSan just by following the above logic. I was actually surprised at first: I expected it to crash and burn, but it didn't. So if I can do it, then anyone can do it too.
Same here; I built a few non-trivial things that passed the first attempt at tooling (valgrind, UBsan with tests, fuzzing, etc) with no UB issues found.
Completely agree. It can, and I think it's extremely annoying that it wasn't.
So we have the next best thing: builtins and flags. So long as those cover all the undefined behavior there is, we can live with it. Compiler gets to be "conformant" and we get to do useful things without the compiler folding the code into itself and inside out.
The point of my article is that this is not possible. This cannot be our end state, as long as humans are the ones writing the code. No human can avoid writing UB in C/C++.
It's honestly not that difficult to be rigorous. The things you mentioned in the blog post are pretty obvious forms of degenerate practices once you get used to seeing them. The best way to make your argument would be to bring up pointer overflow being ub. What's great about undefined behavior is that the C language doesn't require you to care. You can play fast and loose as much as you want. You can even use implicit types and yolo your app, writing C that more closely resembles JavaScript, just like how traditional k&r c devs did back in the day under an ilp32 model. Then you add the rigor later if you care about it. For most stuff, like an experiment, we obviously don't care, but when I do, I can usually one shot a file without any UB (which I check by reading the assembly output after building it with UBSAN) except there's just one thing that I usually can't eliminate, which is the compiler generating code that checks for pointer overflow. Because that's just such a ridiculous concept on modern machines which have a 56 bit address space. Maybe it mattered when coding for platforms like i8086. I've seen almost no code that cares about this. I have to sometimes, in my C library. It's important that functions like memchr() for example don't say `for (char *p = data, *e = data + size; p<e; ...` and instead say `for (size_t i = 0; i < n; ++i) ...data[i]...`. But these are just the skills you get with mastery, which is what makes it fun. Oh speaking of which, another fun thing everyone misses is the pitfalls of vectorization. You have to venture off into UB land in order to get better performance. But readahead can get you into trouble if you're trying to scan something like a string that's at the end of a memory page, where the subsequent page isn't mapped. My other favorite thing is designing code in such a way that the stack frame of any given function never exceeds 4096 bytes, and using alloca in a bounded way that pokes pages if it must be exceeded. If you want to have a fun time experiencing why the trickiness of UB rules are the way they are, try writing your own malloc() function that uses shorts and having it be on the stack, so you can have dynamic memory in a signal handler.
> It's honestly not that difficult to be rigorous.
Ok, let's try it. I pointed GPT 5.5 at the smallest part of cosmopolitan as I could find in two seconds, net/finger. 299 lines.
describesyn.c:66: q + 13 constructs a pointer that can point well beyond the array plus one element.
C23 6.5.6p9:
> If the pointer operand and the result do not point to elements of the same array object or one past the last element of the array object, the behavior is undefined
Now… you may be trolling, but I do feel like this disproves your assertion. Not you, not me, not Theo de Raadt, can avoid UB.
> the compiler generating code that checks for pointer overflow.
Do you need to check for that specifically? What pointer are you constructing that is not either pointing at a valid object correctly aligned (not UB), or exactly one past the element of an array?
Do you mean for the latter, in case you have an array that ends on the maximum expressible pointer address?
I'm a bit unclear on what you mean by "pointer overflow". From mentioning 56 bit address spaces I'm guessing you mean like the pointer wrapped, not what I pointed to in cosmopolitan, above?
Ok, to be clear that it's not just that one type, if you forgive that one:
net/http/base32.c:64: read sc[0] even if sl=0. I assume this is never called with sl=0, so could be fine.
net/http/ssh.c:355: pointer address underflow? Should that be `e - lp`?
net/http/ssh.c:209/229: double destroy of key. can this code path have non-null members, meaning double free? Looks like it, since line 207 does the parsing and checks that parse worked.
net/http/ssh.c:123: uses memset, which assumes that it sets member variable pointers to NULL (per my post, depending on that means depending on UB), and later these pointers are given to free(), so that's UB.
I won't look deeper into net/http, but presenting just the possibly incorrect remaining comments from jippity:
- ssh.c:211 and parsecidr.c:44: length-taking APIs use unbounded strstr() / strchr(), so explicit n with non-NUL-terminated input can read beyond the buffer.
- tokenbucket.c:77 and tokenbucket.c:92: x >> (32 - c) is UB for c == 0 and for out-of-range c.
- isacceptablehost.c:68: long numeric host labels can overflow signed int b before the function eventually rejects/accepts the host.
> For most stuff, like an experiment, we obviously don't care, but when I do, I can usually one shot a file without any UB (which I check by reading the assembly output after building it with UBSAN)
Does this depend on the project, or part of a project? I'm wondering how far that scales, I don't know labor intensive it is -- maybe you can just look at the output and see that nothing funny is happening?
Sure, maybe don't bet your entire company on mountains of Zig code just yet, but aside from the breaking changes it's been perfectly usable and suitable for every project I've ever wanted to work on.
If someone is switching from C because it's too easy to trigger undefined behavior, picking one of the few other not memory safe languages is missing the point.
That’s a taste matter. Being recalled that what is expressed is always depending on some technical details on every move, this is great when one is loving technical details and have all the leisure time to pay attention to them. This is going to be hell compared to sound defaults for someone willing to focus on delivering higher order feature/functionality which will most likely work just fine.
Unedefined behaviour means "we couldn’t settle on a best default trade-off with fine-tuning as a given option so we let everyone in the unknown".
It isn't 1970 anymore. You can get 32-bit ARM MCUs with tens of kilobytes of flash and multiple kilobytes of RAM for less than 10 cents.
We've long since reached a point where chips are cheap enough to be disposable. They are included in paper transit tickets and price tags. There is basically no market left where your volume is small enough that custom application-specific ICs aren't an option, but your volume is large enough that the cost of a few additional kilobytes of memory isn't massively outweighed by the developer time saved.
Want several megabytes of RAM and flash to run Java? That's the price of a cup of coffee!
You always could find deep niche where any high-level technology is not suitable.
I don't think you will program such device in C, rather in assembly, right? When you have like memory for 500 commands, it is easier to go directly to assembler, anyway, with such hardware as a target you don't need portability, this code is 100% hardware-dependable, at it is perfectly Ok.
BTW, which uC your have in mind when you talk about single-digit nA draw (in running state? in deep sleep?), because old 8-bit architectures typically are designed for older node processes and not as energy effective as new one, and draw in sleep doesn't depend much on RAM or FLASH size or architecture, it is more design philosophy.
Anyway, PIC16LF (20nA in deep sleep) or 8051 clone (50nA in deep sleep) or STM8 (~0.30 uA in halt) or ATtinys (100nA in deep sleep), which are covered by "768 bytes of flash and 64 bytes of RAM" description are comparable with EFM32 ARM32-M0+ (20nA in deep sleep), same with uA/Mhz, but ARM32-M0+ will do much more work for each Mhz, so it will be more efficient in the end (faster does all work and go to sleep again).
> Because the last time I looked it appeared to need some godawful slow bytecode interpreter that took up thousands of kilobytes of RAM.
Did you looked at java 1.2 at 1998 last time? Because after that there is compiler which produce some very efficient profile-guide-optimized code and do tricks like de-virtualization which is not possible with static compiler with support of multiple compilation units (like C++).
Really, there was time in history when HotSpot-compiled JVM bytecode was faster than everything that gcc could produce for comparable tasks. Yes, now this gap is reversed again, as both gcc and clang become much more clever, but still gap is not very wide now.
You know what JIT means, right? It means that is is not compiled from the start and indeed runs on a bytecode interpreter until the JIT compiler kicks in.
The java JIT has produced sufficiently fast code for all but the most demanding of HPC applications for going on 20 years. I realize keeping up with new developments can be difficult but the out of date java performance memes are entirely ridiculous by now.
Meanwhile half the world appears to run on cpython of all things.
My life for a browser that doesn't jitter and tear when scrolling or a terminal emulator that can actually process data near the speed my hardware can handle.
Yes, the JIT compiler compiles code. Yes, the results are good. That does not change the fact that the JVM still has and uses a bytecode interpreter, which the comment I replied to disputed.
> -Denial: "I know what signed overflow does on my machine."
Or you just not skip the introductory pages, that tell you what the language philosophy of C is, and why there is UB. Yes, UB can be a struggle, but the first four steps are entirely unnecessary. It means that you do not actually understand the core concepts of the very same language you are using, which is kinda stupid.
I think the issue has been that the line between de-jure and de-facto behaviours has shifted over the years as compiler optimizations suddenly began relying on de-jure intrepretations of UB to increase performance while ignoring de-facto usage of the language.
When that started happened people became alarmed (oMG UB iS TeH BAD!) and since some old UB machines still had industry support (of organisations that actually participated in ISO meetings instead of arguing online) there was never any movement on defining de-facto usage as de-jure and the alarmist position became the default.
Personally I think the industry would've benefited from a Boring C (as described by DJB) push by people that would've created a public parallell "de-jure" standard that would've had a chance to be adopted by compiler creators.
> I think the issue has been that the line between de-jure and de-facto behaviours has shifted over the years as compiler optimizations suddenly began relying on de-jure intrepretations of UB to increase performance while ignoring de-facto usage of the language.
I guess I am too young, and also too much a purist, because I start from the impression of what the language is, not what the implementations happen to do.
> Personally I think the industry would've benefited from a Boring C (as described by DJB) push by people that would've created a public parallell "de-jure" standard that would've had a chance to be adopted by compiler creators.
Mark down is great because it doesn't define a bunch of things. Headline? Its a headline, no font, no sizing, no colors... Just a headline. It means that it can be displayed on any device, printed on any paper, work with any accessibility tool and optimized for what ever requirements the reader has, not what ever the writer thought looked good. The web is full of great content being made hard to access because of poor/inflexible layout choices. Just give me the text and let me choose how to make it readable. The added fact that you can read raw markdown without even parsing it makes it even better. Not having total control over what its going to look like for the reader is a feature not a bug.
> Its a headline, no font, no sizing, no colors... Just a headline. It means that it can be displayed on any device, printed on any paper, work with any accessibility tool and optimized for what ever requirements the reader has, not what ever the writer thought looked good.
God, remember when that was that goal of HTML and the web?
Problem with that is that the default browser styling is extremely ugly and the ability for custom style sheets was removed from the browser GUI many years ago. ReaderMode and Addons can help, but as long as the default is essentially broken and unsupported that whole approach remains a dead end.
On top of that come issues like the lack of pagination support in browsers, which make long document impossible to read and practically require to add custom UI inside the website itself.
ePub works much better, with readers giving control over line spacing, font size, pagination and proper markup for TOC and other metadata, but despite ePub being based on xHTML, browsers have ignored it (only old Edge supported it for a little while).
On this planet, humans have read HTML without parsing for years. People building their first websites without any significant technical knowledge stole HTML by reading the source of other sites and edited it by hand.
Oh, please. Don't insult everyone here by pretending you actually believe HTML is a human readable format like markdown. It was never designed for that and has never claimed that.
It is. Humans do read it, and have read it. Like any language it's just a matter of familiarity.
HTML was designed for humans to read and write long before Claude or compiling everything from typescript or whatever, when websites were all written by hand. In text editors. Even if you were using PHP and templates or CGI you wrote that shit by hand, which meant you had to be able to read it and understand it. Even if you were using Dreamweaver, you had to know how to read and write HTML at some point. WYSIWYG only gets you so far.
Is HTML more difficult to read than Markdown? Sure. It is impossible? Not even remotely. Teenagers did it putting together their Geocities websites.
You can be as snarky as you like, but facts are facts.
I don't think they were appreciating that HTML could be read unrendered. I think they meant that it was up to the browser to render HTML with sensible but unspecified or otherwise user-specified styling (the browser is supposed to be a "user agent", remember?) before web designers started aiming for pixel-perfect control through CSS.
> Mark down is great because it doesn't define a bunch of things. Headline? Its a headline, no font, no sizing, no colors... Just a headline. It means that it can be displayed on any device, printed on any paper, work with any accessibility tool and optimized for what ever requirements the reader has, not what ever the writer thought looked good.
You know, like when you write <h2> in HTML or \section{} in LaTeX?
Whenever the lawnmower thing comes up, I try to also mention dtrace. As far as things to be remembered for, they make some strange bedfellows... although it's better than anything I've managed so I guess congrats.
Hey friend, check the user name of the person I'm responding to (and perhaps check out the people responsible for dtrace and larry ellison lawnmower comparisons). I might appear more coherent afterwards.
For whatever it's worth (perhaps not much?), I was actually asked about this three-decade-old post (!) recently on the Peterman Pod[0], which allowed for a slightly more nuanced discussion.
I really can't think of a better way to respond to this situation. It is clear to me that over the next decade the amount of people who will have been hot-headed kids on the internet who grow up to fully-fledged adults who have said they no longer agree with things in ways that are not kind is going to be a lot higher. I've no doubt said things that I no longer agreed with that made sense in the context of when they were posted.
Thank you for being a good role model and setting the example that saying "that was bad, here is the context, but I don't like that I said that."
Oh! This is a great explanation, thanks. I remember your original exchange (and
I found it baffling and uncharacteristic), and I remember the William Shatner SNL Trek convention sketch, but I never made the connection between them.
I’m currently moving my personal VPS to Oracle Cloud (for a couple of reasons). The new machine’s host name is lawnmower. I have never been so decisive and satisfied in naming a computer.
Just a normal haiku pattern and waiting for tests to finish, 5-7-5.
Could be 'wrong' since erupting is "technically" 3 syllables - but I think it sounds better said fast. Becomes a 7-7-5 that way. Same issue with 'lawnmower.'
oh my god… we thought anthropomorphizing “the computer”, was the problem, when it was anthropomorphizing the principals all along… (yes, i know that is the joke you made, but it was so incredibly appropos that i felt the need to comment to register my amusement / sadness / ¿)
This, "If you have enough, why does it matter if someone else has more" argument doesn't really hold. Yes we can make more TVs and phones so that everyone can get one, but then get to things like centrally located housing, where there is a limited supply. It matters a lot if you want someplace to live, and there are thousands of very rich people trying to out bid one and other for the same space in your city. This is why housing is no longer affordable.
not that I disagree with you in principle but it is a bad example. there are a lot more places where rich do not want to live than places where they do so rich outbidding you for housing wouldn't make even a top-100 list of issues. the "If you have enough, why does it matter if someone else has more" is basically "rich get to live in Monaco, you get to live in Elmo, Kansas" with all your needs met
A good location means more opportunities. Someone located in the center of a large expensive city will have a lot more opportunities to make money and meet people who have influence than someone in Elmo. A business set up in Elmo will not make as much money as a business set up in Monaco. This means that the best opportunities are reserved for people who need them the least.
San Francisco was the cheap to live city that allowed poor, non-monetary obsessed youth to move there and live a hippie lifestyle.
Silicon valley was backward cheap farmland that allowed students in the nearby universities to stay in an area with their college friends and start their business ideas instead of moving back home.
Anywhere there is excess energy/synergy the rich move in and try to capture it, sucking it out. You need places where society can grow, where excess energy is allowed to create excitement/progress/try new things.
Thanks for the shout out. I had no idea my 2h video, without a camera 8 years ago would have such legs! I should make a new one and include why zero initialization is bad.
Thank you for recording it! :) It hits the right balance between opinionated choices with explanations and a general introduction to "post-beginner" problems which probably a lot of people who have programming experience, but not in C, face.
Yeah the idea i that school is somehow a bastion of meritocracy is misguided.
Academia is better at setting clear requirements and measuring those goals, but whether these requirements have anything to do with being successful or useful in the real world is an entirely different matter.
School isn't reality, its mostly not even trying to simulate reality. School breads a lot of "Why was I not rewarded? I did everything they said i should do" disappointment in the real world.
This is awesome. I run a team that uses software I produce and i have a rule that i can’t deliver breaking changes, and i cant force migrations. I can do the migration myself, or i have to emulate the old behavior next to the new. It makes you think really hard about releasing new APIs. I wish this was standard practice.
Sounds like an easy workaround would be versioned APIs then. Missing that, it sounds like the API will forever be stuck, or add-only, creating a mess. That is, if it is not already very stable.
When ever i see "never implement your own...", i know i want to implement it myself. People say that about hard things, and I only want to do hard things. Nobody wants people who can do easy things, people want people who can do hard things. The only way to learn how to do hard things, is to do hard things, so do the hardest things.
So go ahead, write your own date library, your own Unicode font rendering, compiler, OS, game engine or what ever else people tell you to never do because its hard.
By all means, write it. Just don't use it. These warnings are almost always in the context of code you're going to release, not exercises in learning on your own.
Hard disagree here. Use it. Of course, if you running code that drive a pacemaker or a train maybe be careful, but in general, do things. We don't want a world where only three old bearded guys can write a compiler or a physic engine. Do the same errors again and you'll learn, eventually you'll do better than those who were here before you.
Well don't do it and instead of using an off the shelf library that is known to work while the rest of the development team isn't reinventing the wheel.
Doing it for fun and education is fine of course.
What IS the right way to model dates in a pacemaker ...? I hope the answer is "just don't do it" -- but I don't know what reasons there might be for a pacemaker to need to depend on calendar dates in order to best do its job ...
Well naturally it will need to connect to your phone via Bluetooth for the app to proxy update downloads and historic location data uploads. But in order to do anything on the network securely you need an accurate clock and the ability to parse datetimes because the PKI implementation depends on that.
Then the app pings you to remind you that your premium subscription will be expiring soon after which your heart rate will be limited to 100 bpm or less.
Hopefully no real pacemaker manufacturer is allowed to do that. Creating radiation in the middle of a body near a critical organ without a medical reason sounds like a really dumb idea.
Also you want the pacemaker to be as air-gapped and simple as possible, because it needs 100% uptime.
In the case of date libraries, I think if I ported the tests from a few well-known libraries to my own, I'd have reasonable confidence in my own.
Having said that, I don't think date libraries are hard, I think they're messy. Mostly because humans keep introducing convenience fudges - adding a second here, taking eleven days off there, that kind of thing.
You might be right, I haven't checked. It just seems on the face of it such an easy thing to test. Scalars go in, scalars come out. (This could just be me doing the Dunning-Kruger thing).
You could run a fuzzer against two libraries at the same time to find discrepancies....... hmm. That might actually be a good exercise.
Most well-known date library systems have failed in places. Quite a few, still do. So whilst you might get some known regression to test against, nothing can give you a foolproof guide.
You can have reasonable confidence that here there be dragons, but not so much that your assumptions about something will hold.
I'd say write it, probably don't use it, and don't share it unless it's substantially better than the alternative.
This way, you'll learn about it, but you'll more likely stay with something standard that everyone else is using, and you don't share yet another library that wastes others' time and your own (having to read about it, evaluate it, use it, and the migrate off of it when it's abandoned).
This is such nonsense. All the stuff that we use, someone wrote. If nobody makes them, then how is that going to work?
The messaging here is that you should be careful about using what you build on your own because it:
- hasn't been battle tested
- likely has bugs
- isn't mature
The only way that it will be all of those things is if someone invests time and energy in them.
From an ecosystem perspective this is absolutely the right thing. You want duplicate projects. You want choice. You want critical knowledge to be spread around.
> If nobody makes them, then how is that going to work?
I see it as “Dont write your own X, unless you want to maintain X. Here be dragons, this problem is deeper than it appears, the first 80% will be easy, the next 15% will annoy you, and the last 5% will consume your life for weeks, months, or even years. Or you could use a library”
The latter, if you want to get it completely right. I occasionally read the commits in the Qt framework, and from that, I can tell you that date-time stuff is complicated, and not in an instructive way, but in a super tedious way.
This assumes, that the practices/methods used to create a working library are suitable for solving the problems. They might be ill-advised and include tons of workarounds for bad design decisions. Too often following the advice of never reinventing anything (and possibly doing better), is how we ended up with stacking broken stuff on top of other broken stuff, limiting us at every turn with leaking abstractions and bad designs.
It is very possible to have a clean implementation with good design choices overtake an established in time, enabling more extensibility, modularity and maintainability.
An example: People still way over-use regexes for all kinds of stuff. Even in code editors people for syntax recognition, where people really should know better.
In order to have these mature libraries, someone hat to start building them. They all had to to be incomplete, immature and horribly buggy early in their lifetime, too.
Yeah, so do you want to go through that process of shipping broken crap and fixing user complaints one at a time until it happens to work for everyone, which is a mandatory process for all new libraries in one of these areas to go through, or would you rather stand on the shoulders of someone who's already done it?
You assume that you always have a mature option available. That's (a) definitely not a totally generalizable assumption and (b) my point is that mature options only exist because the people that developed them just did it when confronted with the task.
We are specifically talking about something that does have a mature option available. That’s why it’s stupid to try and implement your own version of something complex.
If you change the story such that the product is actually needed and universally immature, of course building it is a valid argument.
Regarding b: Right, and the point of this article is that for those types of things, go for the already-mature thing. You’re arguing a point nobody is making.
I'm not changing any story here. All this "use a library" advice to juniors isn't as universal amd fast as everybody makes it sound. I always find that it sounds too discouraging. There is no magic in these libraries and in their creation - just time and effort. And that's something that needs to be mentioned, too. You can roll your own if you're willing to justify the effort. Depending on the context, it may just not be a good use of your time.
That's why it's also important to point out that no significant piece of code is mature and stable from the start, but was brought into this state iteratively using tools and processes that are available to everybody else, too. The biggest difference between existing mature code and new code under any good development process
is age.
I think there is missing point in this discussion.
Most of the time you build something else.
Like if you build a todo app and have to deal with scheduling you don’t spend time making date library because it’s not your goal. But people would do that.
Heck most developers instead of starting blog on a blog platform start writing code for their own blogging engine.
I think that advice makes sense in the context of cryptography, where the consequences for getting it wrong can be quite serious indeed. I don't think it holds true for something as unimportant as a date parsing library.
Correct date handling (including parsing) can be monumentally important. Imagine an app that reminds people when to take their medications, for example
1) Dates are often stored as strings, so parsing them correctly is a necessary component of storing them. Also, those dates need not be simple app state. They could come from an API provided by your doctor/pharmacy
2) Many people (especially the elderly) take enough medications on different schedules that managing them all would be a significant cognitive load for anyone
It’s just an illustrative example, though. My point is getting dates right (including parsing their string representations) often matters quite a bit. If you disagree, let’s argue about that rather than quibble about the minutiae of the example
Some things are good hard, the kind of hard that's driven by an interesting domain, going deep with well-architected tools or systems, learning lots of cool stuff.
I expect datetime-adjacent code is basically the opposite of all of this. All the hard parts are driven by fiddly adherence to real-world geography, politics, physics/astronomy, etc. There's no underlying consistency from which a sane model can be extracted, it's just special cases and arbitrary parameters all the way down.
I'm up for a challenge of course, but all else being equal, I'm happy to leave work that is the "bad hard" to others.
Reminds me of this passage from Postgres documentation:
”As an example, 2014-06-04 12:00 America/New_York represents noon local time in New York, which for this particular date was Eastern Daylight Time (UTC-4). So 2014-06-04 12:00 EDT specifies that same time instant. But 2014-06-04 12:00 EST specifies noon Eastern Standard Time (UTC-5), regardless of whether daylight savings was nominally in effect on that date.
…
To complicate matters, some jurisdictions have used the same timezone abbreviation to mean different UTC offsets at different times; for example, in Moscow MSK has meant UTC+3 in some years and UTC+4 in others.”
Parsing datetimes indeed sounds like a challenge in collecting, knowing and maintaining all these warped out standards and compromises. ”Bad hard” is a great description
Correct. It's not hard, just stupidly time consuming to the point of being unable to ever produce anything that works 70% of the time.
I hate anyone who will attempt to craft their own 10-lines line parser and then ignore that it fails 4 times a day. Just use the damn library. Thank you.
Write it for fun, but don't ship it. You're wasting everyone's time with your craft.
I get wanting to do hard things, but do you write in binary? Do you crank your own electricity?
My most valuable resource is time. Sure, I could learn more low-level aspects of my craft ... and sometimes I find it useful to do so.
When I focus on doing the hardest, already solved things by re-implementing them my own way, what value am I adding?
I've never met a client who cared about a library or how I did something in code - until it broke. Then, they didn't care who wrote it, they just cared it started working again.
People have built tables but I still build tables myself. Not as many people will use them as people who use IKEA tables, but that’s okay, I’m still going to build them.
I mean, a table is as hard as you make it. I work with rough construction lumber, and make nice finished goods, my point was that people still do stuff that isn’t worth their time financially.
The entire process is "the thing". In the case of a table by adjusting the inputs to the process you can cover quite a wide range of difficulty and required time.
For example, start from a felled tree, use only hand tools, and assemble using medieval joinery techniques. Building a table that way is quite hard by modern standards.
Now if you'll excuse me I need to get back to writing this date parsing library in assembly.
Not even doing it from scratch is the hard thing, the hard thing is getting the experience to know how to fit stuff together the best way to achieve your design and utilitarian goals, what wood to use, etc.
The hard thing isn’t building the date parsing library in assembly, it’s learning assembly well enough to do it in the first place.
I’m not sure where this discussion began but I was rebelling against everyone who says “just buy it” in regards to anything hard to do.
I can't believe this is such a controversial take. Solving hard things by yourself is growth. I 100% agree, rather solve a hard solved problem yourself than learning yet another JS framework or launching yet another revenue losing SaaS ("successful" because of VC). Or whatever. Push hard boundaries.
Nobody is really saying not to build these things. They’re saying the problem is exceedingly annoying to solve—and often not in a technically interesting way but in a way that is just massively tedious—and a better alternative almost certainly already exists.
If you want to build it to scratch an itch, go ahead. If you want to build it for fun, go ahead. If you want to build it because an existing solution gets something wrong and you can do better, go ahead (but know that it is a way bigger undertaking than you might assume at first glance).
The real advice is “don’t casually build your own X”, but that’s less punchy.
An exemplary one is "don't build your own timezone database"
It's not interesting, it's not fun, it's just a process of getting complaints it's wrong in edge cases and then fixing them, over and over until no one can find another broken edge case.
You can start by writing down England is +0, Germany is +2, etc... someone's going to mention DST and you'll put in a field for switching on the Nth Sunday of month X... later you'll run into a country that uses a different rule and you'll add a bunch of spaghetti code or write a Turing-complete DSL, etc... One day someone tells you about a village where they count 17 hour days on seashells and then you'll give up.
And if your DB doesn't produce identical results to the Olson DB in all cases then you created an incompatibility anyway. Might as well just use the Olson DB.
I think it's a spectrum and most fall somewhere on the line, hopefully dependent on the project.
My personal limit is rolling my own crypto, but I'm definitely more on the DIY scale because I agree. It's a fantastic way to grow and learn, and it's likely you might not have the energy to do it outside of work.
It's controversial because 1) good on someone for wanting to do something difficult and 2) I cannot think of a worse thing to try to implement. Maybe trying to parse the world's postal and street addresses is a close second?
> The only way to learn how to do hard things, is to do hard things, so do the hardest things.
and i don't want to pay my employees to learn, i want to pay them to produce output i can sell.
Doing hard things are good, if this hard thing has never been done before - like going to the moon.
Doing hard things which has been done, but just not by you, is not good unless it's for "entertainment" and personal development purposes - which is fine and i encourage people to do it, on their own dime. Like climbing Mount Everest, or going to the south pole.
But if you are doing a project for someone else, you don't get to piggy back your personal wants and desires unrelated to the project on to it.
Except making employers do only easy things will make them stagnate. People who do nothing but simple CRUD apps over and over won't even be particularly good at making CRUD apps... whereas the guy who builds an Unicode font renderer in his free time always seems to write better code for some reason.
Getting better at your job is not just a "personal want" but very much something that the employer appreciates aswell.
Of course reinventing the wheel isn't good in corporate because the reinvented wheel is buggier than the ready made npm package but employers should go out of their way to find hard problems to solve that they can pass to their employees. It's called a growth opportunity.
You can’t convince an employer with that attitude. They’re gonna keep exploiting their employees and “encourage” them to do their “personal development” in their free time.
Unless you work for enterprise consulting where employers appreciate replaceable cogs that they randomly drop into any project, and nicely out project budget regardless of delivery quality.
> and i don't want to pay my employees to learn, i want to pay them to produce output i can sell.
This can be a bad local optimum. It probably depends on what exactly your business does, but it can make sense to pay an employee to acquire knowledge and skills that are needed in the business. You can't buy this off the shelf in all circumstances. Of course, it also has to make economic sense and be viable for the company. Unfortunately, I often see employees doing things quite badly that they don't really understand because they are not given the opportunity to learn properly. I can't imagine that this burns less money in the medium and long term than giving paid employees adequate space to learn.
I am in a work environment where I actually get to do hard shit for fun, learn a ton, and also "get stuff done" and my employer is happy.
For some of the stuff that has been done already, it might still make sense to do your own implementation, for example if you want to be able to experiment without having to navigate and learn a huge codebase and then have to maintain a fork just to have your own stuff in.
Another project we are starting now involves replacing software which is outright crappy and wastes our time. Thankfully my employer was able to see and understand this after talking it through with them.
> Your customers will pay more for things that are hard to do. Ask ASML.
What a silly example. ASML is valuable because it does something no one else does. It's not because it's hard, it's because they have the know-how and infrastructure to do it whereas others don't.
Juggling is hard. Do you know any millionaire jugglers?
No one else does it, because it is hard I thought? Hard to get all the steps and processes aligned to produce what they do. It is so hard, that there is no rich guy that wants to throw money in the hat and do it himself.
Let's be a little charitable and assume they mean just learn. There are hard tasks you can learn from that also provide something you can't just get off the shelf, rather than just reimplementing the wheel.
> People say that about hard things, and I only want to do hard things.
That's perfectly fine. Your time, your hobbies.
> Nobody wants people who can do easy things, people want people who can do hard things.
No, not really. People want people who do easy things, because they are clever enough to avoid needlessly wasting their time having to do hard things when they could have easily avoided it.
It's your blend of foolish mindset that brought us so many accidental complexity and overdue projects. There's a saying: working smart instead of working hard.
> So go ahead, write your own date library, your own Unicode font rendering, compiler, OS, game engine or what ever else people tell you to never do because its hard.
> When ever i see "never implement your own...", i know i want to implement it myself.
Doing stuff for learning is useful, and the intent behind this general phrase is to not ‘implement your own’ something which is both hard and critical in a production environment. I work in cryptography (for security purposes) and have implemented quite a few things myself to learn, but I still use stable, field tested, and scrutinized crypto for any actual use.
> People say that about hard things, and I only want to do hard things. Nobody wants people who can do easy things, people want people who can do hard things.
Only wanting to do hard things limits yourself quite a bit: what about things which seem easy but could be improved? I worked in a non-tech related medical manufacturing job for a bit and took time to learn the process and tools. Afterward, I implemented a few tools (using what my coworkers (who have no programming or IT experience) have available to them: Excel and the VBA on the lab computers) to help them prep inventory lists which they have been doing by hand. Doing it by hand took them 3 hours as a group (and the first shift had to do this every morning), which my tool did in 5 seconds with a single button click. They still use it to this day, about a decade later.
This wasn’t something ‘hard:’ I glued a few files together, grouped a list by a filter, sorted the groups by a column, and made a printout which was easy to read and mark on as they went about their day. However, my coworkers didn’t even know this was possible until someone came in with a different skill set, learned what they did (by doing the job well for months) and then made a solution.
You must be careful with doing only ‘hard’ things. It requires other people to identify what is hard! In addition: crackpots do only hard things and believe they find better solutions than what exists so far (without consulting or learning about what has been done). Interesting people learn about things as they are (with the humility of knowing that they are not experts in most things) and tries to improve them using the knowledge they already have.
Don’t waste your time rolling your own crypto when you could do the _actual_ hard thing and identify unaddressed space to make careful and considered improvements.
It's all about the nuisance created by human behavior. Calendar, DST, timezone, all the problems you never imagined can happen and can only be met in real life scenarios, and you will meet same problem again, struggle then found out the same problem have been solved long time ago by mature library, and the solution doesn't require any smart or advanced technique, just another corner case.
Firstly because I have a great imagination, but secondly because I am old and have a lot of real life scenarios to think about.
State-of-the-art here has changed a few times in my professional career: Once upon a time most time/date libraries used a single integral type and try to make it do double-duty by being both interval and absolute (whatever that means) time by taking the interval from an epoch.
Relatively recently however, that's started to change, and that change has been made possible by people using languages with better type systems reinventing the date/time approach. This has led to fewer bugs, and more predictability with regards to calendar operations in different programs.
But bugs still happen, so this approach is still unsatisfying. One thing I keep having to worry about is distance; I record RTT as part of my events, since when I am looking for contemporaneous events, the speed-of-light actually tends to be a real factor for me.
So I don't think this is solved simply because my problems aren't solved by existing libraries, and I keep getting into arguments with people who think GMT=TAI or something dumb like that.
It's not "all about" anything: Nobody knows shit about what's happening in the next room over, and if there are 12 different date/time libraries now, I guarantee there'll be a 13th that solves problems in all of them, and is still incomplete.
I think in the case of the article the date library isn't necessarily hard but tedious. They mention most date libraries suffer from supporting too many standards or allow ambiguity.
I agree with you though, do the hard things even if it doesn't work 100% right you will have learned a lot. In university I had to implement all of the standard template library data structures and their features, it wasn't as robust as the actual STL but the knowledge of how those work under the covers still comes up in my day to day job.
There things which was a result will make your mind click to an other way to comprehend a problem space and how to navigate through it.
And there are things which are hard due to pure accumulation of concurrent conventions, because of reasons like coordinating the whole humanity toward harmony with full happy peaceful agreement of everyone is tricky.
Handling date is rather the latter. If you dig in the lucky direction, you might also fall into cosmological consideration which is a rabbit hole of its own, but basically that's it: calendars are a mess.
I find this a perplexing comment in view of the fact that almost all of the linked article is in fact about how the author wrote his own date parsing library; the "never do it" bit is just a couple of lines at the start and so far as I can tell is mostly there for fun.
(In particular, at no point does the article actually argue for not writing your own date parsing library. It just says, in essence, "Never do it. I did it. Here's what I did and why.")
Missing context is - there is always something else you work on like the guy was making Eleventy so it was waste of his time.
If you work for a company and build todo app most likely it will not be beneficial for you to implement in-house library because there will be stuff that will bring much more value.
Like you don't have now 2 years to cover for all hard stuff because you have to make synchronization of tasks between devices and your boss most likely won't appreciate that.
"Never roll your own cryptography" is always used in context of building another application it is never "don't become a cryptography specialist".
> So go ahead, write your own date library, your own Unicode font rendering, compiler, OS, game engine or what ever else people tell you to never do because its hard.
You can absolutely do these things. What you need to be aware of is that in most cases maintaining these things to a production quality level is full-time job for a talented engineer. So you shouldn't attempt these IF:
- You have a higher-level aim you are also trying to achieve
- You need a production quality implementation
If one of those isn't the case then knock yourself out.
In a scenario where a programmer has to do this for work and might naively think that date handling is simple, the title is invaluable advice. It is one of those things that can cause real trouble.
OTOH writing, e.g., your own renderer could cause some funny display at worst and maybe some unnecessary effort.
Software companies make money by providing value to their customers via the software they provide. How does reimplementing a hard but already well-solved problem align with their goals? How does that compare with solving a hard problem for which there are no good solutions yet?
The only way you understand X is by making your own X and trying to support it for a few decades, and our industry needs more people who understand X; fewer who just ask chatgpt/stackoverflow/google for "the answer".
Writing an OS. I've learned more about computers, hardware, CPU design, compilers, etc. that have translated into literally every other facet of my IT world than I could have done without this project.
Due to my work I rely on web scraped data for cybersecurity incidents. For Amazon Linux, they are disclosed with the fvcked up US datetime format (Pacific Time) and not in ISO8601 formatted strings which could imply Juliet/Local time.
In 2007 there was a new law that changed when Pacific Time enters/leaves Daylight Saving Time. Instead of making this fixed by a specific Day of a specific Month in numbered form like say "YYYY-03-01 to YYYY-10-01", they literally wrote the law quoting "first Sunday of April" to "last Sunday in October". Before 2007 it was "Second Sunday in March" to "first Sunday in November".
I'm not making this shit up, go ahead and read the law, come back and realize it's even more complex for other timezones, because some nations seem to make fun of this by going to +14:00 hours and -11:30 hours depending on the president's mood on Christmas or something.
In order to find out the Day of a specific calendar date, there's this cool article about Determination of the day of the week [1] which is quite insane on its own already. There is no failsafe algorithm to do that, each method of determining the day of the week has its own tradeoffs (and computational complexity that is implied).
Then you need to get all Sundays of a month, count the right one depending on the year, map back the date to ISO8601 and then you know whether or not this was daylight saving time they're talking about. Also make sure you use the correct local time to shift the time, because that changed too in the law (from 02:00LST to 03:00LDT and 02:00 LDT to 01:00LST before, to 02:00LST to 03:00LDT and 02:00LDT to 01:00LST after the changes).
Took me over 4 fvcking weeks to implement this in Go (due to lack of parsers), and I hate Amazon for this to this date.
PS: Write your own Datetime parser, this will help you realize how psychotic the human species is when it comes to "standards". After all this I'm in huge favor of the Moon Phase based International Fixed Calendar [2]
Reporting of cybersecurity incidents are easily late by a month or more, time zones are well below the rounding error. You will be more accurate to display it as YYYY±6month.
You seem to be not aware that there are a lot of legal obligations which come with providing this kind of inaccurate data. Especially if things go wrong because of it.
Yes, Objective C is built on some deep C calls you can call directly. I was a part of a project that built an automatic wrapper generator for C. Check it out at https://felixk15.github.io/posts/c_ocoa/
