…at least if you let these things autopilot your machine.
I haven’t seen a great solution to this from the new wave of agentic IDEs, at least to protect users who won’t read every command, understand and approve it manually.
Education could help, both in encouraging people to understand what they’re doing, but also to be much clearer to people that turning on “Turbo” or “YOLO” modes risks things like full disk deletion (and worse when access to prod systems is involved).
Even the name, “Turbo” feels irresponsible because it focusses on the benefits rather than the risks. “Risky” or “Danger” mode would be more accurate even if it’s a hard sell to the average Google PM.
“I toggled Danger mode and clicked ‘yes I understand that this could destroy everything I know and love’ and clicked ‘yes, I’m sure I’m sure’ and now my drive is empty, how could I possibly have known it was dangerous” seems less likely to appear on Reddit.
The solution I go for is, don't ever run a coding agent on a general purpose machine.
Use a container or VM, place the code you're working on in the container or VM and run the agent there.
Between the risk of the agent doing things like what happened here, and the risk of working on a malicious repository causing your device to be compromised, it seems like a bad plan to give them access to any more than necessary.
Of course this still risks losing things like the code you're working on, but decent git practices help to mitigate that risk.
I really wish these agentic systems had built in support for spinning up containers with a work tree of the repo. Then you could have multiple environments and a lot more safety.
I'm also surprised at the move to just using shell commands. I'd think an equally general purpose tool with a more explicit API could make checking permissions on calls a lot more sensible.
Superficially, these look the same, but at least to me they feel fundamental different. Maybe it’s because if I have the ability to read the script and take the time to do so, I can be sure that it won’t cause a catastrophic outcome before running it. If I choose to run an agent in YOLO mode, this can just happen if I’m very unlucky. No way to proactively protect against it other than not use AI in this way.
I've seen many smart people make bone headed mistakes. The more I work with AI, the more I think the issue is that it acts too much like a person. We're used to computers acting like computers, not people with all their faults heh.
I don’t think there is a solution. It’s the way LLMs work at a fundamental level.
It’s a similar reason why they can never be trusted to handle user input.
They are probabilistic generators and have no real delineation between system instructions and user input.
It’s like I wrote a JavaScript function where I concatenated the function parameters together with the function body, passed it to eval() and said YOLO.
Sandboxing. LLM shouldn't be able to run actions affecting anything outside of your project. And ideally the results should autocommit outside of that directory. Then you can yolo as much as you want.
The danger is that the people most likely to try to use it, are the people most likely to misunderstand/anthropomorphize it, and not have a requisite technical background.
I.e. this is just not safe, period.
"I stuck it outside the sandbox because it told me how, and it murdered my dog!"
Seems somewhat inevitable result of trying to misapply this particular control to it...
If they're that unsafe... why use them? It's insane to me that we are all just packaging up these token generators and selling them as highly advanced products when they are demonstrably not suited to the tasks. Tech has entered it's quackery phase.
If chainsaws, plasma cutters, industrial lathes, hydraulic presses, angle grinders, acetylene torches, high-voltage switchgear, forklifts, tower cranes, liquid nitrogen dewars, industrial centrifuges, laser cutting systems, pneumatic nail guns, wood chippers, arc furnaces, motorcycles, wall outlets, natural gas stoves, pressure cookers, ladders, automobiles, table saws, propane tanks, swimming pools, garbage disposals, mandoline slicers, deep fryers, space heaters, extension cords, bleach/cleaning chemicals, prescription medications, kitchen knives, power drills, roof access, bathtubs, staircases, bicycles, and trampolines are that unsafe… why use them?
If all those things suddenly appeared for the first time on a Tuesday afternoon, like to many people how LLMs did, then there will be a lot of missing fingers before we figure out what kind of protections we need in place. Don’t get me wrong, the industry is overhyping it to the masses and using the wrong words while doing so, like calling an arc welder “warmth at the push of a button”, but it’s still useful for the right situation and with the right protective gear.
I've been using bubblewrap for sandboxing my command line executables. But I admit I haven't recently researched if there's a newer way people are handling this. Seems Firejail is popular for GUI apps? How do you recommend, say, sandboxing Zed or Cursor apps?
…at least if you let these things autopilot your machine.
I haven’t seen a great solution to this from the new wave of agentic IDEs, at least to protect users who won’t read every command, understand and approve it manually.
Education could help, both in encouraging people to understand what they’re doing, but also to be much clearer to people that turning on “Turbo” or “YOLO” modes risks things like full disk deletion (and worse when access to prod systems is involved).
Even the name, “Turbo” feels irresponsible because it focusses on the benefits rather than the risks. “Risky” or “Danger” mode would be more accurate even if it’s a hard sell to the average Google PM.
“I toggled Danger mode and clicked ‘yes I understand that this could destroy everything I know and love’ and clicked ‘yes, I’m sure I’m sure’ and now my drive is empty, how could I possibly have known it was dangerous” seems less likely to appear on Reddit.