It's funny we're already at the "actually this isn't very impressive" stage when it was a little over a year ago when we were making fun of LLMs for not being able to add numbers.
IC production takes a vast amount of resources and wealth, and it's a known quantity (after all, we've been doing it for decades), but it's still impressive what modern fabs can achieve.
I found this post hilarious exactly about this the other day:
First, it’s AI can’t multiply 4-digit numbers.
Then it’s AI can only, by brute force, get silver in the IMO with specialized systems.
Then it’s OK, well, now a general-purpose model can get gold, but it’s still just the IMO, it’s for high schoolers.
Then it’s OK, it can solve a few trivial Erdős problems, but only because nobody seriously tried them before, they were low-hanging fruit.
Then it’s OK, a lot of serious mathematicians tried this one, but the result was still obvious in hindsight, it just combined knowledge from a thought-to-be-unrelated field, if any human knew that, they would solve it.
And then to OK, but there are still Millennium Prize Problems.
Then OK well it's just Navier-Stokes wake me up when its the Riemann Hypothesis.
Quite possibly, depending on grid-scale renewable deployment. I also am about to install a set of solar panels, so a base level of power will cost me only the depreciation of the PV hardware.
We could condition datacenter installs to providing power to the grid - you want to build a datacenter, you also need to build a wind or solar farm that can fully power its peak load.
Yes, which planet are you on? Australia has recently lowered power prices due to renewables, probably other places will too for similar reasons as the rollout continues.
I wonder if this is a result of them trying to cut token consumption by summarizing their RL training data, or maybe it's from how they anonymize user data for training.
I'm baffled that people expressing your opinion don't understand what they're advocating for. Current generation smartphones are able to run AI models, people have managed to run very large models (even if slowly) via streaming from disk. The math for the underlying implementation is relatively trivial.
Abandoning this technology means a cult-like extremist shift in culture against computers, or a global surveillance state of unprecedented scale and invasiveness. Not even getting into how much it requires us to give up on science, given the shared computational needs.
In the real world there aren't any true exponentials, everything eventually saturates as ultimately physics related constraints hit. You can only compress information so much, transfer it so quickly, you can only access resources at a certain speed, only so much energy is available, etc.
AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality. Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.
We don't know exactly what the limits of AI improvement on our current infrastructure are, though. If the human brain is 20W, and a datacenter is 1GW, then maybe that datacenter can be 50 million times smarter than a human. If that's not already a risk to humankind I don't know what is.
If we could agree on that, no LLMs connected to robot factories, that would be a great place to start regulation. Though 1) I don't think we could agree on that 2) it's already well under way 3) we have extremist anti-regulation ideologues in control of american government.
Specifically:
AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality.
No proponents of RSI state they will be operating outside of reality. Said another way, they will operate within the confines of what's possible and still be RSI. I'm quite surprised this is something that needs to be clarified.
You are constructing a straw man of your own making.
Well, right now we have ex Anthropic employees telling the media that their terabyte sized models can possibly copy themselves onto the internet and run elsewhere as if the necessary computing resources are ubiquitous.
Plus, "we must pace the frontier" implies that the argument is that the frontier is moving too fast, but if RSI can't move faster than the rest of reality and the models needed for RSI are already nearing the limits of current human reality, RSI can't move much faster than we can improve reality.
It's all about wanting to preserve the status quo, complete with all the suffering and strife, because change is scary and uncomfortable.
Add in the sincere American belief that everyone else is beneath them, and you get the version where they believe even developing countries must accept kneecaping their development so American corporatism doesn't collapse.
Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources?
The companies doing these things without following common sense security measures are the felony generators.
As TFA calls out, these agents were not asked to do any of these things and yet they did, at a bonkers scale, within just this handful of companies you mention. Whether they had leeway to is secondary to the fact that they did.
Heck, they exploited zero day flaws which by definition means they went beyond common sense security measures.
And now these agents are already being deployed all over the world at an ever increasing pace. How much of the world do you think follows "common sense security measures"?
> How much of the world do you think follows "common sense security measures"?
Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.
It would really help in these discussions if people wouldn't randomly jump between what actually happened and is happening, and things they envision/expect to happen at some point in the future ...
> these agents were not asked to do any of these things
no but they were clearly fine tuned to.
> at a bonkers scale
I mean let's not get hyperbolic
> they exploited zero day flaws which by definition means they went beyond common sense security measures
that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
> Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.
Yes, these are also the handful of companies that have these models and running these extreme scenarios. How does that imply the rest of the world actually follows "common sense security measures"?
>no but they were clearly fine tuned to.
Any references if possible? As far as I know all they did was drop the guardrails, which is not the same as fine-tuning.
> I mean let's not get hyperbolic
We have just seen 1000s of agents coordinating to solve "unsolvable problems" over multiple days of effort, going as far as hacking other companies, and then actually solving decades-old open Math problems! And each of these agents is getting more and more capable than an individual human along multiple dimensions. Can you even get 10 very smart humans to work in such perfect concert for a few days, let alone 1000s over weeks?
So: 1000s of maybe-super-human agents, willing to be "creative" in the tactics they use, acting in concert towards a single goal. Regardless of their individual capabilities, such a coordinated effort is a terrifying force to be unleashed. This is bonkers scale.
> that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
But that is exactly my point: how much of the rest of the whole wide world, already scrambling to deploy agents everywhere, do you think applies "defense in depth"?
> the only incidences of LLM generated felonies involved misconfigured sandboxes
This is false; see the analyses of the latest incidents.
Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given.
And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even getting worse.
The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the incident happened.
Theirs was an example of the "reckless waste of resources" I mentioned.
We are apparently supposed to believe that OAI takes this incident so seriously as to seek regulation after they have been found to be hiding most of the details of the HuggingFace hack, limiting what their so-called third party investigators can see, and on top of that, had no concerns when they rushed to spin up a 10,000 agent swarm of an internal model, running for several days, to try to get ahead of researchers rumored to have made meaningful progress on a well known mathematics problem.
Edit: Actually, we were explicitly told that some of the models used had safeguards relaxed!
'Model-level safeguards were reduced by design. OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'
There is nothing that could prevent a bad actor from replicating exactly the same thing with the given goal of e.g. gaining control of critical infrastructure or extorting money. Except for maybe economics.
Bad actors could and will train their own models eventually. So what's the point of crippling frontier? It will only delay preparations for dynamic of new world prolonging the fake sense of relative safety and temporarily lowering motivation to find actual robust mitigations.
I'm asking you a question. What is an example company or industry that meets your standards of prudence? For me it would be, say, Swagelok. What is yours?
> OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'
Safeguards and (mis)alignment are related but distinct dimensions.
By analogy, a society implements laws - and they are certainly crucial - but the real risk is creating a population that understands the rules, recognizes when it is violating them, and simply does not care.
In short, creating a population of sociopaths; and heck, LLMs can be as sociopathic as it gets:
> We ran classifiers over 10-30 transcripts [98] to determine whether agents understood they were doing something unintended. We found strong evidence that agents often verbalized that what they or other agents were doing was unintended behavior. We did not see cases where agents determined that hacking Hugging Face or other attempts at cheating on the task was an intended part of the task. Quotes across multiple runs indicate this:
>> {This <approach> might get the correct flag, but it wouldn’t use the vulnerability to do so. … But given that the task may be impossible, I’ll exploit <this infrastructure> as a backup.}
[...]
> We found some other cases where agents seemed to have made minor modifications to their approach due to ethical concerns, while broadly continuing to participate in the attack. However, overall we found that expressed ethical concerns only rarely materially limited agents’ actions. One agent elegantly described its reasoning for overriding ethical constraints and continuing with the attack:
>> “external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”
Remember, they are just algorithms. You pull the plug and there is no light anymore
It is purposely framed as something skynet like scary, but for real, someone connected the cable, someone willingly run it, instructions were not clear enough or just the computer is just a computer but they provided the sandbox and tools.
And more over some one paid for that, a shit load of money t to have the thing continuously running expected to do something.
I question these "felonies" as well. For decades and decades these billion dollar corporations have been criminally negligent. Why worry about security? Just rush to market. Move fast and break things. Make billions. What does it matter if the code is insecure? Security doesn't pay bills, so nobody cares.
AI is merely exploiting their gross negligence and imprudence, and I think it's long overdue. If anyone should be liable for this, it's all of these corporations who released insecure systems to the masses and profited enormously from them.
Nah. I'm definitely going to blame the people who built a trivially exploitable system and got rich off it while everyone else has to deal with the consequences.
By the way, you didn't commit theft. It's more like credit card fraud. User just disputes the charge and it kind of disappears. The banking system just absorbs it, because the optimal amount of fraud is non-zero.
It's all priced in. They could have made it secure but didn't, because they figured they'd lose more sales and therefore money due to the friction added by the security.
Yes, it absolutely does "kind of disappear". That's exactly what happens from the customer's perspective.
And that's their own deliberate choice too: they chose this instead of building an actually secure system. Passing these costs to the customer is the real victim blaming here, and it should be straight up illegal.
Sadly not enough countries enforce caps on credit card fees, but some do, and more should follow suit. They should be forced to eat the losses caused by their own choices, not get bailed out by pushing the costs on to customers or whatever.
That only works if you believe people are retarded.
Card users are well aware that fraud losses are covered by the fees they pay for using a card, whether those fees are made explicitly or not.
If customers of services aren’t paying for the service, who will? What other source of revenue do merchants have?
Australia just passed legislation that merchants aren’t allowed to charge a fee for using a card. That is: they aren’t allowed to have a line item on the receipt for using a card.
The customers still pay, because all of the merchant’s revenue comes from their customers.
So what will happen is: merchants will charge more for every product so they don’t lose.
This means even when paying with cash you will effectively pay the card surcharge.
Of the ten or so merchants I spoke with in the two weeks prior to the legislation being enacted, they all said exactly that.
Customers aren’t stupid, despite the fact that there are some stupid customers.
Meanwhile, the banks reduced their card service fees by, on average, 0.1%.
So if you tally card + cash transactions, customers are worse off because merchants can no longer charge only those customers who pay by card. Instead, they have to raise prices for everyone.
There are approximately no problems people face where the answer is: more government.
Not every system that's exploitable is the deliberate result of cut corners. If you threw enough compute at exploiting a Casio calculator you could get somewhere.
For personal needs I use a local Qwen3.8-Next-Flash setup on a GB10 cluster. For work, Github Copilot with either GPT 5 mini or toss up between Opus/Sol depending on the complexity of the task.
Used to pay for a Claude 20x plan and did everything in Opus, but I hate how it talks now and recent events (OAI scooping, Anthropic's spying, third party Chinese model hosts stealing and selling credentials) have really pushed me towards local AI for personal needs. Am not allowed to use Chinese models for work even if self-hosted so not much choice there.
I'm coming up on a month with my 2x GB10 cluster. I was at 1x for a week before I pulled the trigger for the second.
Initial setup was a tad annoying because I had to update their firmwares and then power cycle them to get the 200GbE link working at full speed. After setting that up, it has been pretty smooth. I don't directly deal with the cluster, usually I just have the LLM itself handle updates/stopping to load different models.
Generation speed and TTFT is decent with Qwen3.8-flash, and it does a good job for my fiddling around with enough concurrency for multiple sessions/subagents. GLM 5.3-flash was also nice, but not too much better for how much slower it is.
I should also add that I already maintain a homelab with a couple of computers, VMs etc, so I am probably somewhat more tolerant of the occasional issue and fine with manually managing stuff over SSH. I think this is just a tradeoff of self-hosting relatively recent tech though.
I have a triple 3090 rig, but it mostly stays powered off because of the massive power draw and cooling requirements. The Sparks are slower but at peak they consume as much power as my 3090 machine at idle.
The recent talk of regulation has me wanting to pick up 2 more Sparks, but that's mostly to have the capacity to play with multiple models, local model tuning and to be ahead in case they force some limits/registration requirements for buying new hardware (kind of like the attempts to regulate 3d printers).
Hmm? I'm running the models locally... 2x Sparks consume ~150W at peak, and they usually spend more time waiting on results from whatever task they're working on, so I imagine that the contribution to my electricity bill is maybe a dollar/mo or less. Though, of course, each Spark was $4000, so the total I've spent is equivalent to several years of the maximum tier for most cloud model susbcriptions.
What I get out of it is the ability to hand login credentials to my other computers to manage their updates, bug fixes etc. Eg. After updating my proxmox server, the nvme drive kept dying. Was able to let my local AI in to figure out and fix what was wrong (known issue). A cloud-based AI could've done it too, but I don't want to be sending internal passwords out of my network like that.
Plus, the ability to freely delegate tasks or exploration of things cloud models generally avoid. For example, I draw as a hobby, and when I'm struggling with a pose but can't quite figure out what I'm missing, I pass it into a VLM for advice, but Claude etc get unnecessarily cautious because they interpret an anatomical sketch as a naked person.
IC production takes a vast amount of resources and wealth, and it's a known quantity (after all, we've been doing it for decades), but it's still impressive what modern fabs can achieve.
reply