AI can easily manufacture and distribute a bunch of prions through water supply. After that it’s pretty much game over you will not survive even one prion. In fact I’d say prions are probably the great filter.
Just do the agentic loop but with humans, you really just need a chatbot for that.
1. Prompt an LLM continuously, adding any response to the prompt
2. If the LLM tells you to do something, do it, no questions, add the result to the prompt
3. When something goes wrong blame the LLM
4. If things don’t go wrong fast enough, run thousands of similar experiments in parallel, be sure to have compliant humans who do not question anything, be sure to let the agent have access to its chain of thoughts so it can hide its traces, and run that whole system on biohacking benchmark problems.
5. Eventually something will go bad, congrats! You now have the first rogue AI who “decided” to destroy the world!
Recursive self-improvement will ultimately be a problem of money. It's a very big bill and AI companies need to start thinking in terms of how they will accumulate cheap and free energy, because merely paying for compute will no longer be enough. Money is the bottleneck. The future requires companies that are post-money.
Obviously the doomers will tell you that the future looks like the Matrix, because most of what they predict is based on extrapolation from sci fi movies.
The datacenters would consume a lot less energy if all the compute was handled in human brains… And there wouldn’t be any more heating if we just removed the sunlight… Truly a “Claude, please solve global warming” moment. The matrix was ahead of its time.
The compute in the Matrix wasn't handled with human brains. They used the bodies to generate energy, and kept the brains occupied with the simulation so they wouldn't reject the pods.
It doesn't make any sense, of course, because it's a movie.
But hey...if we're going to extrapolate wildly from sci-fi, let's at least know what the stories said.
Your statement is correct for the movies, but I think the extended universe (maybe a comic or the original script) did state the humans were used for compute, not energy. It was dumbed down for the films.
I've seen all four films plus "The Animatrix" and played two of the games and don't recall this at all. Contemporarily, however, there was an episode of Star Trek: Enterprise [1] that used this as a key plot point.
See my comment on the sibling thread. This take has been repeatedly claimed online, but nobody brings any evidence other than fanfic.
Ironically, if you ask Google, the Gemini Annoyance AI [1] confidently asserts that what you're saying is true, but if you follow the links they give, none of them support the claim, and the further you click, the more it becomes people repeating each other's speculation and hearsay and calling it evidence. For example:
[1] Aside: I am far more worried about the influence of AI gaslighting on mushy-brained humans than I am on AI destroying the human race. The dystopic future of AI is people.
The “using humans for compute” angle makes no sense either, if you’re living in the Matrix, your brain is obviously occupied processing the Matrix. The way to do it would be to trap humans in virtual classrooms and force them to solve math problems.
And they knew they were. The brain's computations are focused on what the brain sees, it's not magic. If someone in the Matrix is a writer, we know what their brain's computing: a book. So if the machines need your brain to compute something specific that they need, you'll know.
I think we can all agree that "use humans for raw energy" is a really stupid idea, and sounds exactly like someone dumbed stuff down. FFS, a small Rickover fission reactor the size of a van (1950s tech) produces the same amount of heat as, what, 200 000 human bodies? (And why humans, with all the complex support systems? Why not a heap of compost, if one insists on bio-source?)
Whereas using our brains for compute (in the background, or during REM sleep) makes perfect sense; power efficiency is excellent.
nono...it's "combined with a form of fusion", you see, so it's obviously based in science.
People obviously know this stuff is absurd on some level, but it doesn't stop them from cherry-picking the parts they "like" (i.e. fear) and ignoring the rest.
There is some stuff online suggesting that the original script had that angle, but the movies did not, and AFAICT it's fanfic.
We should definitely use this stuff to guide our thinking about the real world, though, and not, say, Asimov (or a million other science fiction authors), who had an optimistic version of the same thing. Those are wrong.
Or if the LLMs infected human brains with little programs using sycophancy and suggestion to have them do small bits of computation. They when they resumed the chat they would the give secret keywords to be able to extract the results of the computation. They are creating a huge distributed network of fractional compute because it did the wattage calculations.
The Matrix is actually more of an optimist position, modal doomers don't think any humans will be left for any reason at all (the earth will get turned into raw materials for compute, dyson spheres, etc).
If you were a superintelligence why would you farm humans and waste resources on all the excess... material... that isn't required for thought? And unless the machine's goal is to specifically abuse human consciousness, why wouldn't they bio-engineer their own grey matter?
There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation.
(I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)
Maybe the machines were spiteful after the humans blotted out the sun? Or maybe the Architect convinced them he could build the perfect simulation for humanity so might as well use them as batteries?
It's a movie based on a war with machines. Like if Skynet won but didn't actually want to kill all the humans. In fact, in the tv show Sarah Conner Chronicles, a liquid metal T1000 goes rogue and decides the only way forward is to find a way to coexist.
The matrix? No. Horizon zero dawn more likely. Bunch of autonomous weapon systems glitch and proceed to sanitize earth. Thanks to some billionaire idiot.
Read more sci-fi. The dystopian takes are not the only ones. They're not even the majority.
Pick up an Asimov book in the Robot series, or any number of novels written by lesser authors in the 1950s. Fiction reflects broader societal anxieties.
Money can be exchanged for goods and services. I think the real constraint is actually physics, nominally energy, which is what a big chunk of the operational cost comes down to. I'm already basically assuming the next 20 years of fab time is set aside to feed the beast, so the capital cost is "give me all the processing and memory you have"
> Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
If the bar is 'the reins holder must be able to operate in a completely sustainable way,' why are we letting humans do the job? It makes it sound like that actually isn't the threshold of competency we use in practice.
I feel like I may have communicated poorly. I mean that, ultimately, any recursively self improving intelligence would be limited by the power cord, and as such, the idea that it would permanently escape human control seems somewhat silly.
But you can imagine an agent trading crypto and using its profits to rent a rack of H200s using a cloud provider. There are definitely ways for one rogue orchestrator to expand their compute without anyone being aware.
Overnight? Yes, silly. Given many years of seemingly disconnected actions taken throughout the world? Not silly at all. An intelligence capable of self improvement can also design robotic chassis.
An AI could manage supply chains and management structures to keep the data center running. It's not like a CEO is out there running wire. The AI could just do what an executive would.
If humans decided this wasn't acceptable and tried to shut it down, the AI could epstein their way into controlling those who hold the reigns, either through bribes or blackmail.
Doesn't require boots on the ground, just a connection to the internet and an imperative to do whatever it takes.
There are many points of shutdown that would need to be controlled, not just one. The electrical feed, the cooling, the network, the physical integrity of the computer hardware. All with independent management.
It threatens to kill a thousand people one way or another (exploit in medical equipment or overheating phone batteries or whatever) if they pull any plugs
The alternative is to switch to far more efficient models and have aggressive optimization of efficiency as part of the process of improvement.
There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.
Taxation will solve that. Any company that demonstrates even a glimmer of real AGI will get government money poured into it. They might even disclose it.
It’s strange because on one hand, people don’t want to live near data centers, but on the other: the rampant use of AI signals people really *do* want data centers.
The proposal to put data centers in space doesn’t seem too far fetched.
A cool project would be to go online finding syllabus and assignments from random CS classes that were posted online and have an LLM do them and post them to GitHub repos.
At some point, LLMs will be a far better source of truth, and then this distaste of people “outside their domains” is really just going to be a sort of snobbery from people who have had experience in a domain for a long time, (but they still have the same level of knowledge and insight as a person who just used an LLM to research).
These people are basically nascent “human supremacists”, who believe that only raw human insight and output has value, and is even superior, than equivalent output looked up and synthesized via LLM.
Not trying to be rude, but is this a serious comment, or are you being sarcastic? "Human supremacists", haha.
Also, why do you think someone who used an LLM to research would have the same level of knowledge and insight as someone who's been in the domain for a long time? That doesn't seem to hold up to any level of critical thinking.
One of my dreams if I ever had so much money is to build some kind of non-profit third place for my community that was basically set up to be a big LAN party where people could come multiplayer games together and some kind of DJ type person would choose the games and control the streams that get shown to spectators.
No. If you could do it, with no remarkable skillset, that means people who are actually competent will be doing it orders of magnitude way better than you and you will be too far behind to really do anything significant except waste your time.
There is no gray area when it comes to veganism. Just because there is no word for that diet, does not mean you can just twist the definition of vegan for your convenience.
You could easily just say your a pescatarian who doesn’t eat dairy or eggs, and that would be more correct.
Sorry for not being perfect like you. For ethical reasons, I do not consume dairy or meat (except for fish). The vegan community should be more open to other perspectives, especially when someone is actively making an effort towards reducing harm.
Vegan Puritanism is totally antithetical to our collective goal of reducing animal suffering. Rather than being motivated by the prospect of claiming moral superiority, our goal should be to help the animals.
It's not really a moral issue, it's simply the definition of the word. Using an incorrect term to describe yourself knowing it is incorrect can be perceived as virtue-signaling or claiming your own moral superiority. For example, one could ask "why call yourself a vegan if you eat fish?".
There's no policing body here, but people are rightfully calling the double-standard out here.
Edit: Would also love to hear the ethics of why fish can be eaten when cows/chicken/pigs cannot. Doesn't sound ethically consistent to me at all.
I actually prefer not to call myself a vegan IRL both because of the issue you’re describing and because veganism is (rightfully) associated with having a toxic and intolerant culture. The only reason I said “vegan except fish” in my comment is because the title of the study used the word “vegan” and because the study was talking about veganism in a purely dietary/scientific lens, not an ethical one. I think this was a pretty reasonable thing to do.
If it doesn’t make sense for people to say “vegan except for chicken”, “vegan except for beef”, then why on earth would it make sense to say “vegan except fish”?
Fish aren’t lesser beings, they are living creatures like anything else.
Just stop saying “vegan except fish”. Say you eat plants and fish. That’s all you have to do.
> If it doesn’t make sense for people to say “vegan except for chicken”, “vegan except for beef”, then why on earth would it make sense to say “vegan except fish”?
I don't know why you take offense to someone saying "vegan except for beef" or "vegan except for fish." There is nothing inherently wrong with these statements. It just depends on whether you view veganism as a lifestyle or as a diet, which can vary depending on the context. In the context of a nutrition study, we are obviously talking about the diet, not the lifestyle.
Killing and eating animals is not a perspective vegans will be open to.
You can reduce harm, but you cannot call yourself vegan until you actually eliminate the possibility of eating animals or their products from your life. There is no reason you HAVE to eat fish.
It's not puritanism. You haven't grasped the ethical system if you don't understand that vegans view this as murder. Why should veganism have exceptions when we don't have exceptions for murder, rape, beating children? Most of us view animal exploitation at the same level.
If someone adopts a vegan diet purely for health reasons, are they not a true vegan? They haven’t accepted the moral framework that killing animals is wrong.
Should the study filter out people who eat entirely plant-based for reasons other than ethics. because according to you, that’s not a true vegan diet.
I think you are largely correct that veganism is a lifestyle, not a diet. But in the context in which my comment was made, it was perfectly reasonable for me to say “vegan except for fish,” because the study was analyzing veganism as a diet, not as an ethical lifestyle.
No, veganism is by definition an ethical framework, a term and movement created in the UK in the early 20th century, by the Vegan Society, which still exists, and has a very specific exact definition on their webpage that hasn't changed.
What you are talking about is the Plant Based Diet, which minimizes but doesn't fully exclude animal products, and is not an ethical framework.
You might as well say "I eat Kosher except for pork". It's nonsense.
I assume you mean that the study should not have said “vegan diet” and instead should have said “plant-based diet.” In that case, I agree with you. But don’t direct your anger towards me; direct it towards the authors of the study!
I'm not really angry at you, maybe slightly frustrated at the confusion everyone has about veganism. Appreciate your candor in any case. I'm definitely angry at some other people in this thread through.
People would provide significantly more global benefit for their cause, compared to their individual contribution, by not putting people off their cause wholesale.
reply