> Like the x-axis label of the first plot mentions that it shows gridlines every 8 ticks - I don't think that's a choice I've ever seen a person make
The spacing being 8 is important and seems not so bad to call this out in an extra place.
Imagine LLMs didn’t exist for a moment. Is this the part of the article you’d find most intriguing? That the grid lines being an unusual distance apart is written both on the axis and in the description? Would it have seemed inhuman on a blog post about debugging a weird display issue on an e-reader?
> They may be capable of solving decades-old maths problems, but on this particular axis they seem to be floundering at the level of a fresh graduate that's desperate to talk about what they've done rather than what their audience needs to know
Have you ever worked with people beyond graduate level? Ever seen scientific posters at a conference?
> Imagine LLMs didn’t exist for a moment. Is this the part of the article you’d find most intriguing?
I would think it's odd! That's why I mentioned it. I like to pay attention to vis, I feel like I've seen many charts from Tufte-y perfection, to slapdash undergrad presentations, and to experienced consultants banging things together in Excel at 1am. This tic feels very LLM-y to me, which imo makes it interesting to think about - how does it arise? I also think this failure mode has been resistant to model upgrades in a way that mathematical reasoning has not, which I also find interesting as someone in ML research. It feels like this kind of mentalisation or theory of mind is a capability which isn't obviously elicited from the frontier labs' crop of RLVR tasks.
> Have you ever worked with people beyond graduate level? Ever seen scientific posters at a conference?
I've worked with a fair number of people beyond graduate level. In previous jobs I've run teams and hired people - hence my comment about graduates. We used to have an interview stage where candidates presented some simple data analysis, and it often centred on what they had done rather than what was most worth knowing. I have also attended scientific conferences, in fact I was at one a week ago! Is there a particular inference you would like me to make from those experiences?
I want to really understand this, your issue is that the non-standard grid spacing is called out on the axis? The grid spacing being 8 is unusual but clearly sensible here - your issue is just that this is written also on the axis label?
> Is there a particular inference you would like me to make from those experiences?
The incredible amount of unnecessary detail people put into their posters specifically (hence the better poster movement), desperate to show everything.
> I want to really understand this, your issue is that the non-standard grid spacing is called out on the axis? The grid spacing being 8 is unusual but clearly sensible here - your issue is just that this is written also on the axis label?
Pretty much yep - it struck me as odd and reminded me of an LLM tic I've been thinking about recently, so I thought I'd write a short comment.
> The incredible amount of unnecessary detail people put into their posters specifically (hence the better poster movement), desperate to show everything.
I hadn't heard of the better poster idea before, thanks for clueing me in. I largely agree that posters have a lot of unnecessary detail, but I think it's a different flavour to the LLM stuff - more like they're so excited to tell you everything in their paper. Whereas LLMs focus on odd details, or fail to explain things.
Can I ask, if you use LLMs often in your work, have you never run into this experience?
Do you really have a hard time imaging that someone would call out an obvious AI slop label as weird and poor quality? Like it's quite clear people do not like LLM generated content and feel betrayed when they didn't consent into consuming LLM generated content.
If I saw the charts in the pre AI era I would also be confused. I mean it is just awkward and not done like this. OP is right, no one on earth says they put gridlines every so and so ticks anywhere, much less the figure title. Gridlines should be obvious what they are trying to show on a chart like this. They should actually be aligned with the tick marks on the x axis. In fact, why are those on a nice even 10s scale while gridlines are on 8 ticks? If this was undergraduate work I saw at a conference, I'd guess they were hungover when they made it and didn't realize they made the tick marks and axis labels incongruent with eachother, and it should have been 10 marks. I'd still have no clue why they wrote that in the title. I'd guess that they'd never presented a figure like this and guessed what might be important to include.
The fact that we are now in the ai era where figures like this come so easy with no thought is actually even more damning to the article author. How can I trust anything in this article is any good given that they thought these figures were fit to publish? Either they didn't proof the work or they didn't care, and both are pretty bad possibilities.
Worth noting with this that those ~1000 agents were shorter lived things that had to communicate via a package registry cache, access the internet via a 0-day in the package manager and did the HF attack while having to save current state and organisation in a remote sandbox. All while managing using their token limits on the task they were assigned and what else they were doing. I wonder how few it would have required if they were actually tasked with hacking HF and supported in doing so.
They did, they found how to fully cheat, but thought this could be caught so then dedicated time to getting a different cheat and how to hide their transcripts. There is a lot around deciding which agents should/shouldn't fail their own tasks in order to contribute to the group.
They explicitly say that attacking hf is not allowed in the rules though, and the research into how to edit their transcripts doesn’t line up with this either.
Perhaps I’m not being as strict with the word sandbox but they were sandboxed right? They did not have generic internet access they exploited other software to make external requests.
You're right. It's my opinion that if your sandbox has a path to the internet, it is not a sandbox, it's a gimmick.
And the 2 other incidents with OAI/ANT had the same issue, but it's even funnier - sandbox in those cases had a direct access to internet because someone forgot to configure it right.
I've seen very early models do similar things on my machine when they hit some unexpected blocker when trying to access a path. I remember early sonnet opening a file in browser because OS sandbox prevented from accessing it directly.
I've also had models discover a syslog-ng server (that I for some reason had ssh key inside), to get into my unraid server because machine they were running on didn't have direct network connection to Unraid server.
It can't be just me who is aware LLMs have been doing such things for the better part of last 2 years. I probably have better sandboxing on my machines now than trillion dollar companies crying AI will kill us all. That's at the very least, negligence to me.
It depends IMO about how strict this is. It's pretty awkward to refuse to call something a sandbox because it may have an unknown bug that would allow escaping. Or rather in this case it was that they had access to a package manager, and the models discovered a bug that allowed them to access the internet (first they discovered that they could use the cache to leave messages).
I do get your point, I just think an overly strict definition can be awkward too. This wasn't as simple as the sandboxes having internet access and writing "pls no internet calls" in the prompt.
Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
Isn't it simply that there are two competing goals that the LLM received RL for, honesty on one hand (a goal that is often assumed as implicit for humans) and producing a solution that meets expectations (which doesn't technically require honesty)?
So the LLM didn't read and interpret the prompt and decide via discussion to violate ethical behavior, the unethical result merely won out because ethics wasn't a hard requirement (and one that isn't reliably detected in the result). An LLM doesn't fear punishment, so ethical behavior is simply one of many positive signals that were trained into it.
> Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
I don't think those words have a useful enough definition to draw a strict line around them to be honest, and getting into that seems to get massively into the weeds. For me, those neatly encapsulate the behaviour as seen, to answer the questions here about what happened. The models did not seem to be confused as to what the goal was or what the intent was. They did not hack HF because they were told to.
What was the inner state there? How would something not being allowed expressed internally? Maybe such language is one way to elicit certain behavior but not a statement of what was permissible?
I'm referring to their transcripts of the reasoning and output tokens - this doesn't go into the detail of evaluating hidden states as there's also iirc evidence of better models having one internal state but putting something misleading down in the "reasoning" tokens.
The either output or reasoning tokens, or perhaps in the messages they were sending each other on the boards they created, have them saying explicitly that doing these things to HF were not allowed then doing them anyway, or at least not notifying people. What I'm getting at broadly is this was not a case of "we told it to attack however it wanted and it chose to hack HF" or "we told it to attack a simulation but it did the real thing" or "we explained not to do that but it was so far back in the context window the models acted like they never saw it" or even "the instructions were not clear".
Yes, my point was more that I don't know whether parsing those outputs as a human is a useful thing to do or not (even though it is in human language of sorts). What machines mean or want elecit might be different from a human interpretation, especially in relation to any RL "forcing".
There’s definitely issues with using them to understand what the models were “thinking” but we can use them to answer a few questions. Most relevant here is that the idea or instructions that attacking hf would be out of scope was not simply lost in the context.
It wasn’t one agent forgetting things because of context, they explicitly discussed with each other and themselves the problems with going outside of the parameters of the task.
No, the first LLM left a text file that the latter LLMs then read. Since these are memoryless black boxes, any words they happen to pick up along the way is treated as the function to evaluate the output to. There's no fucking collusion here as if it were a rogue hacker group, it's a text predictor that received instructions as it always does and executed those instructions blindly.
You can replace discussed if you want with leaving text files or comments in directory names that other ones then read, if you want, it's just an extremely awkward way of talking.
From my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan.
Yeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet.
Me: How do I do xyz?
Bot: Reads website titled "Doing xyz in abc way"
Bot: As per your requirement to do xyz in abc way ....
These things are borderline useless with web search. It's amazing that they just throw out their entire training data and read you the first three things they found on the Internet.
The attention is shifting towards RL, harnesses, and memory systems from the pretrains of more intelligent and capable base models. So extracting additional capabilities from what we already have.
That is a much easier catch up game. GLM 5.3 and DeepSeek flash 4.1 also demonstrate significant jump in cyber capabilities. So yeah, it is a slowdown in the place where it matters. RL has been around for ages, there's no moat there if you already have a good enough pretrain.
Tell me you haven't tried letting Astra go without telling me.
Astra can confidently one-shot 500k lines of slop, with 800k lines of tests covering it, without testing a single intended product requirement, and none of it actually working.
All models require hand holding. Fable and Astra are no exceptions. The difference is only in the amount of hand holding required, and there's essentially no gap here anymore between American and Chinese models.
I only use Chinese models sparingly because American models are so much cheaper with subscriptions, that it doesn't make economic sense to not use them. If/when that changes, I could simply route to cheapest model that's available at the moment and I wouldn't notice much difference in most applications.
> Like the x-axis label of the first plot mentions that it shows gridlines every 8 ticks - I don't think that's a choice I've ever seen a person make
The spacing being 8 is important and seems not so bad to call this out in an extra place.
Imagine LLMs didn’t exist for a moment. Is this the part of the article you’d find most intriguing? That the grid lines being an unusual distance apart is written both on the axis and in the description? Would it have seemed inhuman on a blog post about debugging a weird display issue on an e-reader?
> They may be capable of solving decades-old maths problems, but on this particular axis they seem to be floundering at the level of a fresh graduate that's desperate to talk about what they've done rather than what their audience needs to know
Have you ever worked with people beyond graduate level? Ever seen scientific posters at a conference?
reply