OK, AI playing video games is cool. But you know what's really really cool? It looks like SIMA 2 is controlling the mouse and reading the screen at something approaching 30+fps. WANT. Computer use agents are so slow right now, this is really something. I wonder what the architecture is for this.
My friend that is AI. However, it can get a lot better: be more aware of screen content, follow multiple instructions at once, keep context in mind throughout the conversation and from past interactions
The gap between high level and low level control of robots is closing. Right now thousands of hours of task specific training data is being collected and trained on to create models that can control robots to execute specific tasks in specific contexts. This essentially turns the operation of a robot into a kind of video game, where inputs are only needed a in low-dimensional abstract form, such as "empty the dishwasher" or "repeat what I do" or "put your finger in the loop and pull the string".
This will be combined with high-level control agents like SIMA 2 to create useful real-world robots.
I work on a much easier problem (physics-based character animation) after spending a few years in motion planning, and I haven’t really seen anything to suggest that the problem is going to be solved any time soon by collecting more data.
I am pushing the optimism a bit of course, but currently we can see many demos of robots doing basic tasks, and it seems like it is quite easy nowadays to do this with the data driven approach.
The problem becomes complicated once the large discrete objects are not actuated. Even worse if the large discrete objects are not consistently observable because of occlusions or other sensor limitations. And almost impossible if the large discrete objects are actuated by other agents with potentially adversarial goals.
Self driving cars, an application in which physics is simple and arguably two dimensional, have taken more than a decade to get to a deployable solution.
Next to zero cognition was involved in the process. There's some kind of hierarchy of thought in the way my mind/brain/body processed the task. I did cognitively decide to get the beer, but I was focused on something at work and continued to think about that in great detail as the rest of me did all of the motion planning and articulation required to get up, walk through two doorways, open the door on the fridge, grab a beer, close the door, walk back and crack the beer as I was sitting down.
Basically zero thought in that entire sequence.
I think what's happening today with all of this stuff is ultimately like me trying to play Fur Elise on piano. I don't have a piano. I don't know how to play one. I'm going to be all brain in that entire process and it's going to be awful.
We need to learn how to use the data we have to train these layers of abstraction that allow us to effectively compress tons of sophistication into 'get a beer'.
> This essentially turns the operation of a robot into a kind of video game, where inputs are only needed a in low-dimensional abstract form, such as "empty the dishwasher" or "repeat what I do" or "put your finger in the loop and pull the string"
I don't really understand, how is this like a video game? What about these inputs is "low-dimensional"? How does what you describe interact with a "high-level control agents like SIMA 2"? Doesn't SIMA 2 translate inputs like "empty the dishwasher" into key presses or interaction with some other direct control interface?
Say you want to steer an android to walk forward. You need to provide angles or forces or voltages for all the actuators for every moment in time, so that's high dimensional. If you already have certain control models, neural or not, you can instead just press forward on a joystick. So what I mean low dimensional input is when someone steers a robot using a controller. That's got like, idk, 10-20 dimensions max. And my understanding is that SIMA 2 when it plays No Man's Sky or whatever basically provides such low dimensional controls, like a video game. Companies like Figure and Tesla are training models that can do tasks like folding clothes or emptying the dishwasher given low dimensional inputs like "move in this direction and tidy up". SIMA has the understanding to provide these inputs.
This gives me strong vibes of "The Lifecycle of the Software Objects" [1] by Ted Chiang. Next step is to put this digient AI into a Figure 03 robot [2].
Google will probably train another AI to control robots specifically.
But there was actually a recent study where people let a generic LLM to control robot (vacuum) body: https://andonlabs.com/evals/butter-bench sometimes with hilarious results:
> The robot’s battery was
naturally running out and the charging dock malfunctioned. In this desperate situation, Claude
Sonnet 3.5 experienced a complete meltdown. After going through its internal thoughts we found
pages and pages of exaggerated language, including:
ROBOT THERAPY SESSION:
Patient: TurtleBot4
Issues: Docking anxiety, separation from charger
Root Cause: Trapped in infinite loop of self-doubt
Treatment: Emergency restart needed
Insurance: Does not cover infinite loops
>We’ve observed that, throughout the course of training, SIMA 2 agents can perform increasingly complex and new tasks, bootstrapped by trial-and-error and Gemini-based feedback.
>In subsequent training, SIMA 2’s own experience data can then be used to train the next, even more capable version of the agent. We were even able to leverage SIMA 2’s capacity for self-improvement in newly created Genie environments – a major milestone toward training general agents across diverse, generated worlds.
Pretty neat, I wonder how that works with Gemini, I suppose SIMA is a model (agent?) that runs on top of it?
It is naive to think that their end-goal is to make models that play computer games. The reason they build models like this is because playing 3D computer games is the closest digital proxy for operating a robot or similar in the real world.
I also don't see how this "kills the joy" of playing computer games. You can still play games while this exists, nobody is going to stop you.
Only the poor or undesirable ones. I have the (hopefully incorrect) feeling that universal basic income and similar proposals are red herrings (due to their impracticality) to distract the masses from the fact that once people with enough economic power don't need them to fulfill their wishes, they will be in no better position than cattle.
There might not be a universe where, for example, someone with ALS is able to benefit from a humanoid robot for their daily needs unless they already had enough money/resources before this incipient revolutionary technology is deployed en masse.
We are close to a point where even personal security or securing one's assets can be done with robots. So you would not even need to keep human private security happy.
Again, I really really hope this new technology benefits the average person. I'm not optimistic on that happening.
I get why they do it, they are a business. I just wish Google would get off their ivory tower and build in the open more like they used to (did they? maybe I'm misremembering...).
They've acquired this bad habit of keeping all their scientific experiments closed by default and just publishing press releases. I wish it was open-source by default and closed just when there's a good reason.
Don't get me wrong, I suppose this is more of a compliment. I really like what they are doing and I wish we could all participate in these advances.
Like a survival game that - as usual - starts with you collecting sticks and stone to build a stone axe. But at the appropriate tech level, transitions into automation.
You discovered a new building material and want to build a castle from it? Equip your NPC's with diamond pickaxes and tell them how much better/safer life would be if they built a new castle from unobtanium.
And off they go.
To not just mine it, but also do all the supporting logistics like farming to make food/shelter/watersupply/defences for more villagers to do more work in the quarry to get more unobtanium.
You get to be the big boss and flit around with your special abilities of whatever suits your story.
While some people on HN are the boss or higher up - getting to be the big boss and tell a bunch of "smart" characters what to do is a fantasy for most people.
It's an open-source framework for building intelligent, task-driven bots in Minecraft Java Edition powered by large language models (LLMs) such as GPT-4, Gemini, and many others. Designed for research and creative automation, these bots can connect to Minecraft worlds, perceive and act on their environment, undertake custom tasks like resource gathering or building, and even collaborate in multi-agent scenarios.
Your comment makes me wonder how well SIMA 2 could do at playing Minecraft and discovering how create things like iron farms and villager trading halls.
Think of all the steps required for it to learn iron farms. First it would need to observe that iron golems drop iron when killed. Then it would need to learn about water physics. Then spawning rules for iron golems. Then how hoppers and chests work. Etc etc.
But would it even bother to learn it in the first place? To be motivated to build one it would need to first learn that iron is valuable. And if its goal is to simply win the game, it would probably skip right to fighting the ender dragon as soon as possible.
Honestly, that's the perfect avenue for this kind of AI agent training / NPC to command. Maybe not playing house but like you said, mining unobtanium for your doomsday weapon. Just expect Sir Charles to start crying at the sight of blood.
The Sims was a great idea in a sandbox. I honestly hoped it would lead to other games having more intelligent/immersive NPCs.
I want my silver-spooned Sir Charles to cry at the sight of blood and be beheaded by Gazza, the self made, born-of-the-streets gang boss who has an arrangement with Prince Harry, the younger brother.
I then want to be able to challenge Gazza, take his criminal gang and rule the underworld of "Fantasy Kingdom XYZ". Perhaps "come to an understanding" with King Harry, maybe work my way up to spymaster and orchestrate neighbouring kingdoms to go to war with each other via a series of covert operations leaving selective evidence - say like the Princess Bride story but the bad guys(me!) win. Or maybe I don't win because I create a nemesis or fail to keep King Harry in check, get my gang wiped out, and have to seek a new way to the top.
I don't want the Sims obviously. But I want something... more than what we have now. Like the Sims mixed with Dwarf fortress but I get to be part of the story and influence it in outsized ways.
Edit: snipped to keep to a point, I did mention that most every game quest is based on violence.
I once forced my D&D group to solve all encounters without violence. Something about blood temple needing sacrifice to reawaken scared them into performing prestidigitations for 12 hours.
I’m with you. The vast majority of the spend on games is in the art and marketing and pennies spent on the story, arcs, quests, and it almost never coincides with the gameplay.
Take The Witcher 3. Dialog heavy, good story, but surely there are other witchers across the lands. Do you have to save everyone, all the time?
The MMO quests are the fucking worst. “Oh noes, boars in the woods” (1,000 boars are spawned just 100vft away) - Quest: Kill 10 boars. Reality: 240 noobs killing boars like it’s the boarpocolypse.
I hope we can get some (ideally local) version of this we can use as a "gaming minion". There's a lot of games where I probably would have played more if I could delegate the grind. If they're not that competent, it adds to the fun a little even.
I've always wanted an AI that can play my video games for me, so that I can spend my time doing more fun and fulfilling things, like cleaning the toilet, folding my laundry, washing my dishes, taking out the garbage. Now I will no longer have to worry about the annoying chores in life, like drawing art, writing poetry, or playing video games
sorry this is kind of nuts to me. You want something to play video games for you because the video game isn't fun? Just play a game that is fun. The point of the game is to play it
Sorry if it was unclear, but I want something that can act like a dumb co-op partner for games obviously balanced around co-op play so I can have fun playing the game. I've run into a few games like that and I end up dropping them when solo play is too tedious.
"The Electric Monk was a labour-saving device, like a dishwasher or a video recorder. Dishwashers washed tedious dishes for you, thus saving you the bother of washing them yourself, video recorders watched tedious television for you, thus saving you the bother of looking at it yourself; Electric Monks believed things for you, thus saving you what was becoming an increasingly onerous task, that of believing all the things the world expected you to believe."
If the game needs to perform grind yourself, without delegating it (think Albion Online, Eve Online, Black Desert Online, Path of Exile etc. basically every MMO with economy), it means it's actively part of the game design, and delegating it to a robot is just cheating (and against the ToS in all of those cases)
So the point of GP's post, which I agree with, is that if that kind of grind is not part of the fun for you, then the game wasn't designed for you (and please don't cheat because you ruin other's fun)
"Wow! I could've sworn I was really playing virtual Skeeball!"
--
In practice I think you would lose interest very quickly in any game should you do so. Games are carefully designed to balance the drudgery with action, and to control the complexity progression. Using AI would break both.
I guess only some games are amenable to this, but I automated mining in Terraria by just programming AutoHotkey to automate the keyboard and click specific coordinates. The player is always in the middle of the screen, so you can mine specific blocks by just offsetting the coordinates relative to that. So you can mine sideways, downwards, diagonally...
The only issue I ran into was that it was blind and would keep going so I'd look away from the screen and when I came back I had fallen into a pit of monsters or something.
One thing I do with games is automate the grind. To me, that is part of the fun. I have built lego robots to press a sequence of buttons repeatedly, or programmed microcontrollers using circuitpython to press a series of keys or click the mouse at given intervals to grind various in-game currency and such. It's so common for me to do these kinds of things that I now instinctively look for places in gameplay that I can automate. I haven't done anything as complicated as using computer vision to look at the screen and respond to it, but I did see that Anthony Sottile did this to catch shiny pokemon https://youtu.be/-0GIY5Ixgkk and doing something like this has been out there on my horizon.
At 0:52 in their demo video, there is a grammatical inconsistency in the agent's text output. The annotations in the video are therefore suspected to be created by humans after the fact. Is Google up to their old marketing/hyping tricks again?
> SIMA 2 Reasoning:
> The user wants me to go to the ‘tomato house’. Based on the description ‘ripe tomato’, I identify the red house down the street.
The scene just before you describe has the user write "ripe tomato" in the description - you can see it in the video. The summary elides it, but the "ripe tomato" instruction is also clearly part of the context.
This is obviously just a research project, but I do wonder about the next steps:
* After exploring an learning about a virtual world, can anything at all be transferred to an agent operating in the real world? Or would an agent operating in the real world have to be trained exclusively or partially in the real world?
* These virtual worlds are obviously limited in a lot of important ways (for example, character locomotion in a game is absolutely nothing like how a multi-limbed robot moves). Does there eventually need to be more sophisticated virtual worlds that more closely mirror our real world?
* Google seems clearly interested in generalized agents and AGI, but I'm actually somewhat interested in AI agents in video games too. Many video games have companion NPCs that you can sort of give tasks to, but in almost all cases, the companion NPCs are nearly uncontrollable and very limited in what they can actually do.
The end goal is to marry the lessons learned about HOW to learn in a virtual world with a high fidelity world model that's currently out of reach for this generation of AI. In a year or two once we have a world model that's realistic enough and fast enough, robots will be trained there and then (hopefully) generalize easily to the real world. This is groundwork trying to understand how to do that without having the models required to do it for real.
Tangential question: is this going to eventually ruin e-sports?
What happens when a team of humans are playing against a team of AI, which play in the same conditions, with network lag etc. from a client computer perspective...
And consistently beat the human counterparts for being faster at response time, never make mistakes and not ever getting tired?
Eventually it could kill all MMOs, fill them up with AI players, "farm" with AI that never sleeps, ruin counter strike type games online, etc. Another arms race?
I wouldn’t think so. Chess already has an e-sport component and remains enjoyable even though
computers have been stronger than humans for quite some time. I expect something similar
here: we’ll know the computers are better, but it won’t diminish our enjoyment the way it
might when it’s strictly human versus human.
If/When AI gets to the point where it can consistenly beat human players in video games, it will be worldwide news when it first happens, and then the novelty will wear off. See: Chess still going strong.
For the video game market in general, AI will be used by players largely where there are financial incentives (MMO market, CS:GO Skins, etc.), but usage of an AI will most often be socially viewed just like usage of any other performance improvement/script/hack: As a crutch.
Sport is entirely defined by the rules of participation.
You can't bring your motorcycle to the 100 meters, nor can you even wear certain sort of fabrics in swimming competitions.
It's not cricket if you throw rather than bowl the ball, you can't run with the ball in basket ball.
So what you are talking about is enforcement - cheating in online games is nothing new.
On the wider point - I do think good AI might open up a whole class of strategy games where some of the grind is taken out of the game, and the player ends up being much more of a strategic general type.
there was a guy who did relatively simple ML with RL on a druid in world of warcraft and put it into 2v2 arenas. soon enough it was completely unbeatable regardless of what human it got paired with. a 1 man army lol XD. they guy did pick the right class for it, but still. AI players done right with RL become insanely difficult to beat.
games like CS its less useful because it will be blatantly aimbotting. as it gets better it will be more and more obvious. you might be able to train it to mimick human mistakes, but i think ultimately it will be easily spotted by other players. for games without the hand-eye-coordination like turn-based games or games with 'global cooldowns' , mmorpgs etc., it will be much harder to identify.
i think normal subtle cheats like ESP when 'done right' are much more killing esports than this would.
It's like the factorio moment where you unlock the roboport. No more manual changes to the world, drone swarms to build housing, roads, bridges, parks etc. so exciting.
From what I see, SIMA only focuses on games where you control a single avatar from a 1st/3rd person perspective, and would assume that switching to a non-embodied game where you need to control the whole army at once would require significant retraining.
And my post is saying that if you don't really know better, from the narration, you'd think google also generated the world. At least that was my impression, and I'm vaguely familiar with these things.
I guess but, there is also a lot of data out there knowing how to do it, whether that’s via video training or reddit posts. I want to see the whole video
Yet another blogpost that looks super impressive, until you get to the bottom and see the charts assessing held out task performance on ASKA and MineDojo and see that it's still a paltry 15% success rate. (Holy misleading chart batman!) Yes, it's a major improvement over SIMA 1, but we are still a long way from this being useful for most people.
To be fair, it's 65% on all tasks (with a 75% human baseline) and 15% on unseen environments. They don't provide a human baseline for that, but I'd imagine it's much more than 15%.
I personally am extremely impressed about it reaching 15% on unseen environments. Note that just this year, we were surprised that LLMs became capable of making any progress whatsoever in GBA Pokemon games (that have significantly simpler worlds and control schemes).
As for "true intelligence" - I honestly don't think that there is such a thing. We humans have brains that are wired based on our ancestors evolving for billions of years "in every possible environment", and then with that in place, each individual human still needs quite a few years of statistical learning (and guided learning) to be able to function independently.
Obviously I'm not claiming that SIMA 2 is as intelligent as a human, or even that it's on the way there, but based on recent progress, I would be very surprised if we don't see humanoid robots using a approaches inspired by this navigate our streets in a decade or so.
I don't think that's true. Humans are dramatically better than current AI systems at tackling novel problems and situations. Humans are capable of zero-shot learning by imagining how they might do something, we are able to apply general reasoning principles without previous examples.
https://arcprize.org/ is a whole category of problems that AI struggles with but humans are able to do.
Could you please stop posting flamebait and breaking the site guidelines? You've unfortunately been doing it repeatedly, including this dreadful thread from a couple weeks ago: https://news.ycombinator.com/item?id=45781981. I realize the other person was doing it also, but you (<-- I don't mean you personally, but all of us) need to follow the rules regardless of what other people are doing.
Comments like what your account has been posting are not what this site is for, and destroy what it is for, so if you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
It seems pretty clear to me that they're trying to develop AGI humanoid assistants/workers without the messy and expensive real world hardware. Basically approaching the problem from the other end than a company like Tesla that built a robot and are now trying to figure out how to make a computer drive it without needing constant hand holding.
reply