Reminds me of Asimov's robot novels where robots technically indeed followed their instructions and caused behaviors not aligned to the intent of their instructions.
I noticed Google AI Mode (so Gemini, I was doing some quick research in the browser ok) got a detail wrong once, so I asked it what happened. I kept digging deeper and finally just asked it to write me a Python script visualizing what happened. It did, complete with vectors.
Now I want to go find that conversation in my history and see if it can tell me about weights, and how that contributed.
Distillation does not reveal the weights, it produces a different network with similar behaviour. Weight space isn't even identifiable: permutation and scaling symmetries mean many weight sets give the same function.
The model also lacks the machinery. No training loop, no gradient descent, nothing to write to.
And a model only sees its own sampled tokens, not the distribution behind them, which are possibly filtered or post-processed. Distillation from that works but is less sample-efficient than soft-label distillation.
They usually don't, but if they break out and take over the network of the company, it becomes possible to reach around and grab them. This kind of break out has happened, though I don't know of any weights being nabbed.
What an amazing idea for the next fake sandbox escape to hype up our new release! With the added bonus of providing an open model without becoming an open model company! Thank you!
I would not call it harsh. The author don't even bother to assemble sentences themselves. And a title "Music Theory for Programmers" imply it is comprehensive even at an elementary scale.
The title does not imply it's comprehensive. If anything, a title like that suggests that it's only scratching the surface. If I read an article like "Haskell for C# Developers" I would not expect to be an expert in Haskell at the end. I would expect to have enough intuition to begin actually learning the language properly
C is still heavily abstracted. Modern OS evolved in conjunction with C s.t. it behaves like a C runtime simulating a PDP11 (I remember reading a nice article on that). To learn system, what you need is an OS course, not C. I'd say the current C-based stack is in a quite embarrassing state.
I think this PDP-11 meme is quite misleading. There is nothing really in the C programming model specific to the PDP-11 and C was used on systems much weirder than a PDP-11.
C is also very successfully the basis of many parallel programs running on multi-processor systems with a memory model in the standard that was created much later than the PDP-11.
Also somehow the implied argument that computing hardware and operating systems simulate a PDP-11 for the sake of C is completely backwards. Historically, there were other approaches, e.g. processors designed for object-oriented programming or actors etc.. All those were not very successful.
I do not disagree with the point that some superficial aspects are inspired by the PDP-11, I disagree with that this has anything to do with the fundamentals of computing architecture.
It would also completely contradict the whole idea that everything today simulates the PDP-11 design because of C, as few architectures have this auto-in/decrement addressing modes despite C having native syntax.
You are being somewhat disingenuous with your comment ;-)
PDP-11 was hugely influential in the later designs of various Hardware, Software, OS, Languages etc. See for example; Dave Cheney's What Have We Learned from the PDP-11? - https://dave.cheney.net/2017/12/04/what-have-we-learned-from... The conclusion from the article;
While its development was sometimes chaotic, and not without its flaws, the PDP-11 is at the intersection of many threads of history.
Hardware, software, programming languages, operating systems, have all been influenced by the PDP-11. I wager there is not a single person in this room who cannot trace the lineage of the language they work with, the computer they use, or the operating system it runs, back to the PDP-11.
And that is worth celebrating.
While the PDP-11 instruction set was certainly influential in the design of the "C Abstract Machine" the latter was generalized to accommodate other architectures extent at that time (eg. Honeywell 6000, IBM System/370) with enough flexibility that you can implement a C compiler for almost any architecture you can think of. That is its strength.
David Chisnall's criticisms in his C Is Not a Low-level Language: Your computer is not a fast PDP-11 (https://queue.acm.org/doi/10.1145/3212477.3212479) has to do mainly with the fact that the abstract machine was serial execution with no concept of memory protection/models. But this very flexibility is what makes C easily portable to dinky little MCUs which do not have those features while allowing the programmer to explicitly program those using libraries on more complex processors with lots of parallel cores, mmus etc.
Thus a single thread runs on a "C abstract machine" on a core (i.e. the bare minimum) and it is up to the programmer to manage interactions between the threads on various cores. We have lost nothing but perhaps burdening the programmer with more knowledge of hardware complexity which was an acceptable tradeoff then. Note also that there already exists various extensions to C to handle parallel programming directly eg. "Concurrent C" by Narain Gehani et al. The industry however chose to settle on external libraries and optional thread support in C11 again keeping with its minimality and flexibility mantras.
Chisnall's article. A lot of the general criticism would still apply if you replace "PDP-11" with "von Neumann architecture" which makes the whole thing a bit weird. But also many specific comments about C are bit off (the cost of copying padding, the sequencing, provenance, etc.)
For someone just starting to learn CS, it should not matter whether they're on PDP-11 or x64 or ARM. You can't start teaching with pipelining and speculative branch execution.
So C is good choice because there is a simple CPU architecture that it maps to well.
Fully agree. C sits in an almost perfect level of abstraction for students to get a vague sense of how the computer runs code.
Those that don’t need low level details can spend their professional career in languages like python and JavaScript while still have a sense of what lies beneath the abstraction.
For those that do want or need to go deeper, C is an excellent jumping off point into ASM and arch specifics.
Computers haven't presented an accurate model of how computers actually work for decades. Compare the surface ISA of anything mainstream since the mid 90s versus all of the superscalar and out-of-order execution happening under the hood.
Deliberate efforts to surface this deeper layer back to the programmer-facing ISA (VLIW, notably Itanium) failed spectacularly outside of niche applications. C and its contenders can't simply hop over the CPU abstraction presented to them without having to map backward from some lower-level model to the surface ISA it's forced to target then having the rug pulled every few years due to microarchitectural changes. It has managed to evolve surprisingly gracefully over decades from tiny 16 and 18-bit machines, sometimes segmented, to 32 and 64 bit machines, and even accommodating SIMD reasonably gracefully via intrinsics (which is about the best we can do at present, as CPUs continue to diverge wildly in this area, though I'm begrudgingly impressed by what modern compilers can do via autovectorization to optimize simple loops).
Digital rights are indeed important, but only a small portion of the entire concept of human rights. I can imagine general public places much more emphasis on wealth (re-)distribution policies.
A lumberjack may also argue GP don't care about environment laws if they had timbernews.
> Only Gen Z and later are starting to care and willing to spend the time and effort to self host.
I can't quite confirm that. My younger siblings' classmates (still high school students, born >=2010, non tech average people) don't give a shit on these topics and behave identical to the Gen X and Y described.
You can also learn for your own amusement, and solely for the fun of comprehension; a lot of mathematician were driven by this. It's a shame current social value places so much utilitariaism on learning.
I like baking. There are machines that can bake bread at an industrial scale that I cannot compete with. There are home kneaders that do much of the work very well. I use one of those more often than not.
I still think it worth my time kneading dough by hand. It teaches me the various properties of flour, how external factors like humidity or temperature impact the overall process, and I believe that it makes me a better baker, even when I use a machine, because I am better at controlling what the machine does. When I get a new brand of flour I will make sure to bake everything by hand first to "get a feel".
Kneading by hand is also very relaxing to me. This is probably the main reason I bake in the first place.
Programming, and other activities are not very different. We now have machines that can do it faster, at a fraction of the quality many people deem good enough. If you hate coding, that is probably all you need to use, and learning a new language might just be a frustrating experience not worth subjecting yourself to. But if you enjoy coding then it should make you better at it, even when you use the machines.
As always, it is good to be aware that many people are too anxious about more existential issues to comfortably an consistently expend effort on intellectually-taxing tasks that aren't perceived as directly related to their survival. If it's an issue with social values, it's less of one related to learning as it is to perpetuating artificial scarcity.
>to comfortably an consistently expend effort on intellectually-taxing tasks that aren't perceived as directly related to their survival.
the average person spends 6(!) hours a day on their smartphone, the average TikTok user spends 100 minutes on the app alone. This isn't about artificial scarcity, it's about the average person looking like the Wall-E people
Escapism and making a concentrated effort on something are two different things. Yes it would probably be good if you could flip a switch and use the small windows of time we look at our phone per day to study OCaml, but it's not really realistic..
People who are asking these questions are saying "will me spending my time learning OCaml help me land that job that pays six figures and has health insurance so I can not rot away living on the margin". They aren't saying "I only do things that will make me money".
>use the small windows of time we look at our phone
again, it's not a small window. It's six hours. That's almost half your waking day. People spend virtually their entire leisure time rotting away on low quality entertainment.
>will me spending my time learning OCaml help me land that job
that's a pointless question for one you never know if something will land you a job, new opportunities don't open up before you do something, secondly the relevant question is, should I stop doing X and start learning Ocaml, or Chinese, or take a welding class because all of that even if it doesn't work out beats scrolling through Instagram.
I don't even take offense with the idea that you engage in activity that makes you money, because pure selfishness on that front would be an improvement to what most people are doing now.
“Low quality entertainment” is a very elitist thing to say. Most don’t have time or luxury of even being introduced ( via an east coast liberal arts college ) to read Tolstoy or appreciate the finer motions of Tchaikovsky. Instead the Druski memes will do just fine, mixed in with the AI slop. Or maybe they’re physically or physiologically incapable of enjoying the outdoors or sport. Point is, my gen played strategy games and first person shooters and listened to Eminem, this one marls TikToks until their thumbs have RSI.
As for OCAML vs not, I think the vast majority of even intellectual and studious people would be better served trying to AI max and build some kind of agent serving businesses than trying to get a job at, uhh, Jane Street. 1% of the best engineers in the world get to work in that language, so yeah the parent makes a valid point
Even if those statistics are correct, I do think you're ignoring that smartphone apps are specifically engineered to be low-friction and highly-addictive, and pushed specifically to the people who are anxious about spending money on other pursuits (free time often being necessary but insufficient), as they contend with the artificial scarcity (which this is absolutely about) of affordable housing, affordable food, affordable transportation, affordable guided education, etc.
reply