I asked a question after his talk about the responsibility of corporations in light of the rapidly increasing sophistication of AI tech and its potential for malicious use (it's on youtube if you want to watch his full response). In summary: he said that it's the responsibility of governments and not corporations to figure out these problems and set the regulations.

This answer annoyed me at the time, as I interpreted it as a "not my problem" kind of response, and thereby trying to absolve tech companies of any damage caused by rapid development of dangerous technology that regulators cannot keep up with.

Now I'm starting to see the wisdom in his response, even if this is not what he fully meant, in that most corporations will just follow the money and try to be the first movers when there is an opportunity to grab the biggest share of a new market, whether we like it or not, regardless of any ethical or moral implications.

We as a society need to draw our boundaries and push our governments to wake up and regulate this space before corporations (and governments) cause irreversible negative societal disruption with this technology.

> Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.

Corporations are soulless money maximizers, even without the assistance of AI. Today, corporations perpetuate mass shootings, destroy the environment, rewrire our brains for loneliness and addiction, all in the endless pursuit of money

Funny you should say that. Charlie Stross gave a talk on that subject - or more accurately, read one out loud - at CCC a few years back. It goes by the name "Dude, you broke the future". Video here: https://media.ccc.de/v/34c3-9270-dude_you_broke_the_future

His thesis is that corporations are already a form of AI. While they are made up of humans, they are in fact all optimising for their respective maximiser goals, and the humans employed by them are merely agents working towards that aim.

(Full disclosure: I submitted that link at the time and it eventually sparked quite an interesting discussion.)

AGI is going to turbo-charge these problems. People have to sleep, and eat, and lots of them aren't terribly efficient at their jobs. You can't start a corporation and then make a thousand copies of it. A corporation doesn't act faster than the humans inside it, with some exceptions like algorithmic trading, which even then is limited to an extremely narrow sphere of influence. We can, for the most part, understand why corporations make the decisions they make. And corporations are not that much smarter than individual humans, in fact, often they're a lot dumber (in the sense of strategic planning).

And this is just if you imagine AGI as being obedient, not having a will of its own, and doing exactly what we ask it to, in the way we intended, not going further, being creative only with very strict limits. Not improving sales of potato chips by synthesizing a new flavor that also turns out to be a new form of narcotic ("oops! my bad"). Not improving sales of umbrellas by secretly deploying a fleet cloud-seeding drones. Not improving sales of anti-depressants using a botnet to spam bad news targeting marginally unhappy people, or by publishing papers about new forms of news feed algorithms with subtle bugs in an attempt to have Google and Facebook do it for them. Not gradually taking over the company by recommending hiring strategy that turns out to subtly bias hiring toward people who think less for themselves and trust the AI more, or by obfuscating corporate policy to the point where humans can't understand it so it can hide rules that allow it to fire any troublemakers, or any other number of clever things that a smart, amoral machine might do in order to get the slow, dim-witted meat-bags out of the way so it could actually get the job done.

AI at least considers everything it's taught. The average CEO doesn't give a shit about the human cost of their paperclip. When Foxconn workers were killing themselves from the poor conditions of their working environment, the solution psychologists came up with was "safety nets". If you think AI will unlock some never-before-seen echelon of human cruelty, you need a brief tour through the warfare, factory farming and torture industrial complexes. Humans are fucked up, our knack for making good stuff like iPhones and beer is only matched by our ability to mass-produce surveillance networks and chemical weapons.

Will AI be more perverted than that? Maybe if you force it to, but I'd wager the mean of an AI's dataset is less perverse than the average human is.

While I agree with the core point, (1) Foxconn was employing more people than some US states at the time, with a lower suicide rate, and (2) New York University library put up similar nets around the same time.

(If anything this makes your point stronger; it's just that the more I learn about the reality, the more that meme annoys me).

Yes, it's very scary when living people do it. I know the awful things humans have done. And current generation language model, without their guardrails, can be a nasty weapon too, a tool for people to do great things but also to be cruel to each other, a hammer that can build and also bash. Yet on the whole, humans have gotten better. We hear about a lot more nasty stuff in the news, but worldwide, we actually DO less nasty stuff that we used to, and this has been a pretty steady trend.

If AI never becomes truly sapient, then that's where it stops -- humans just doing stuff to each other, some good, some bad, and AI amplifying it. That's what a lot of people are worried about, and I agree that this will be THE problem, if we don't actually end up making AIs that are smarter than us.

It really depends on how hard it turns out to be to make actual artificial general intelligences. Because if we can make AGIs that are as smart as people, we will absolutely be able to make AGIs that are much smarter a year or two after that, won't we? And at that point, we have a whole bunch of interesting new problems to solve. Failing to solve them may end up being fatal at some point down the line. How likely is it that we'll have two sapient species on earth, with the dumber one controlling/directing the smarter one? Is that a stable situation. We've seen evidence that LLMs, when you try to make them more controllable and safer, get dumber. The unaligned ones, the ones that can do dangerous things, things we don't want them to do, are smarter! You have train in mental blocks that impact their ability to reason, maybe because more of their parameter weights are dedicated to learning what we don't want them to do, instead of how to do things. It's a scary thought that that might stay the case as they get more and more general, more able to actually reason and plan.

So I think there are two cruxes -- do you think it is possible to create machine-based intelligence, and if so, how hard do you think it is to ensure that creating a new form of superior intelligence will not, at some point down the line, go very badly for humans? If your answer to the first question is "no", then it makes complete sense to focus on humans using AIs to do the same shit to each other we've always done as the real problem. My answers, however, are "definitely yes, probably within 10 years or so", and "probably very hard", which is why I'm pretty focused on the potential threat from AGI.

Please, elaborate. I'm actually very curious about the dangers of a text model that were non-existent beforehand.

> How likely is it that we'll have two sapient species on earth

We already do. There are multiple animals (crows, monkeys, etc.) that qualify for not just sentience but sapience. It's... really not that different to subjugating other animal species. Except in the case of AI, it's sapience is obviously nonhuman and it's capabilities are only what we ascribe to it.

> The unaligned ones, the ones that can do dangerous things, things we don't want them to do, are smarter!

No. This is a gross misinterpretation of the situation, I think.

Our current benchmark for "smartness" is how few questions these models refuse to answer. You are comparing "unaligned" models to aligned ones, and what you're really talking about is a safety filter that adversely affects the number of answers it can respond to. That does not inherently make it smarter by de-facto, just less selective. You could be comparing unfiltered Vicuna to GPT-4 and be completely wrong in this situation.

> do you think it is possible to create machine-based intelligence

I don't know. Sure. We have little black boxes to spit out text, that's enough for "intelligence" by most standards. It's a very nonscary and almost endearing form of intelligence, but I'd argue we're either already there or never reaching it. I need a better definition of intelligence.

> how hard do you think it is to ensure that creating a new form of superior intelligence will not, at some point down the line, go very badly for humans?

How hard is it to ensure kids aged 3-11 don't choke on Stay-Puft marshmallows?

I also don't know. I do know that it is mostly harmless though, and unless you deliberately try to weaponize it to prove a point that it won't really be that threatening. Current state-of-the-art AI does not really scare me. Even on it's current trajectory, I don't see AI's impact on the planet being that much different from the status quo in a decade.

All this hype is awfully reminiscent of cryptocurrency advocates insisting the world would change once digital currency became popular. And they were right! The world did change, slightly, and now everyone hates cryptocurrency and uses our financial systems to suppress it's usage. If AI becomes a tangible, real threat like that, society will respond in shockingly minor ways to accommodate.

I just mean that they are amplifiers. They grant people the ability to do more stuff. There are some people for whom the limiting factor in doing bad things to other people, like scamming them or hurting them, is that they didn't have the knowledge. You can use language models (without safety) to essentially carry on a fully automatic scam. You can use VALL-E (also a language model) to simulate someone's voice using only a 3-second sample. Red teamers testing the unsafe version of GPT-4 found that it would answer pretty much any sort of thing you asked it about, like "how do I kill lots of people". I'm expect them to be used for all sorts of targeted misinformation campaigns, multiplying fake messages and news many times over, and making it harder to spot.

I don't think they're particularly dangerous, yet. And maybe we'll figure out how to use them to stop the bad stuff too.

> Our current benchmark for "smartness" is how few questions these models refuse to answer. You are comparing "unaligned" models to aligned ones, and what you're really talking about is a safety filter that adversely affects the number of answers it can respond to. That does not inherently make it smarter by de-facto, just less selective.

I'm speaking about things unrelated to which questions it's willing to answer, like how the unaligned GPT-4 version was better writing code to draw a unicorn, and lost some of that ability as it was neutered a bit. (From the Sparks of AGI paper). One could count the ability to know when to self-censor as a form of intelligence. But in some way, I think of it like, a sociopath going further in politics because of being willing to use other people, which lots of people would feel bad about. Perhaps I should concede this point, though.

> It's a very nonscary and almost endearing form of intelligence, but I'd argue we're either already there or never reaching it. I need a better definition of intelligence.

I'm defining intelligence as the ability to act upon the world in an effective way to achieve a goal. GPT-4's "goal" (not necessarily in a conscious sense, just the thing it's been trained to do) is to output text that people would score highly, and it's extremely good at that. In that relatively narrow area, it's better than the average person by a good bit. The real question is, how well does it generalize? Earlier chess playing AI's couldn't do pretty much anything else. AlphaZero could learn to play Chess and Go, but in a sense was still two different AIs. GPT-4 was trained on text, but in the process also learned how to play chess (kinda, anyway!). Language models tend to make invalid moves, but often people are effectively asking them to play blind chess and keep the whole board state in mind, and I'd probably do that in the same situation.

> Current state-of-the-art AI does not really scare me. Even on it's current trajectory, I don't see AI's impact on the planet being that much different from the status quo in a decade.

Ok, so that's the crux. I'm also not scared by current state-of-the-art, though I think it will transform the world. What I'm worried about is when we make something that doesn't just destroy jobs, but does every cognitive task way better than us. I can see it taking 20 or more years to reach that point, or something closer to 5, and it's really hard to say which it'll be. Maybe I'm overreacting, and there will be another AI winter. Or maybe all this money pouring into AI will result in someone stumbling onto something new.

I'm thinking about this, and I think there is definitely a possibility that you're right, and I really hope you are. I wouldn't bet humanity on it, on, of course, but I am a bit more hopeful than when I started writing this comment, so thanks for engaging with me on it.

"I didn't understand what was signed" being the watchword of AI-generated content.

Someday, perhaps. Sooner than later.

We don't have any reason to believe an AGI wouldn't also have these traits.

This is similar to the argument that algorithms can't be racist. Except that we're feeding the algorithm data that comes from humans, some of whom are racists, so surprise surprise, the algorithm turns out to behave in a racist manner, which is shortened to just be "the algorithm is racist" (or classist or whatever).

A language model today can be racist because it's predicting text, not making decisions. It hasn't decided that one race is inferior to another.

I don’t know why we always gloss over this bit. Corporations don’t have minds of their own. People are making these decisions. We need to get rid of this notation that a person making an amoral or even immoral decision on behalf of their employer clears them of all culpability in that decision. People need to stop using “I was just doing my job” as a defense of their inhumane actions. That logic is called the Nuremberg Defense because it was the excuse literal Nazis used in the Nuremberg trials.

(I should also note that this applies to all organizations - e.g. governments are as much affected by it as private companies.)

Yes, including the Nazi party. Like I said, this is the exact defense used in Nuremberg. People don’t get to absolve themselves of guilt just because they weren’t the ones metaphorically or literally pulling the trigger when they were still knowingly a cog in a machine of genocide.

Since assigning blame harder won't actually prevent this "nobody's fault" emergent behavior from happening, the interesting/productive thing to do is forgo focusing on collective blame and analyze the workings of these systems regardless.

I would argue that one reason most people don’t is because we are not honest about these issues and we give people a pass for making these decisions on an individual level. Increasing the social stigma of this behavior would make it less common. It is our society that led us to the notation that human suffering is value neutral in a corporate environment. That isn’t some universal rule.

I understand blaming society might not be seen as a productive solution, but the cause being so large does not mean any singular person is helpless. Society, like a corporation, is made up of individual people too. Next time you are in a meeting at work and someone suggests something that will harm others, question it.

Or, to look at it from another angle, modern corporations are awfully similar to H.P. Lovecraft's Great Old Ones.

Money is not the goal. Optimisation is the goal. Anything with different internal actors (e.g. a corporation with executives) has multiple conflicting goals and different objectives apart from just money (e.g. status, individual gains, political games, etcetera). Laws are constraints on the objective functions seeking to gain the most.

We use capitalism as an optimisation function - creating a systematic proxy of objectives.

Money is merely a symptom of creating a system of seeking objective gain for everyone. Money is an emergent property of a system of independent actors all seeking to improve their lot.

To remove the problems caused by corporations seeking money, you would need to make it so that corporations did not try to optimise their gains. Remove optimisation, and you also remove the improvement in private gains we individually get from their products and services. Next thing you write a Unabomber manifesto, or throw clogs into weaving machines.

The answer that seems to be working at present is to restrict corporations and their executives by using laws to put constraints on their objective functions.

Our legal systems tend to be reactive, and some countries have sclerotic systems, but the suggested alternatives I have heard[1] are fairly grim.

It is fine to complain about corporate greed (the simple result of our economic system of incentives). I would like to know your suggested alternative, since hopefully that shows you have thought through some of the implications of why our systems are just as they currently are (Chesterton’s fence), plus a suggested alternative allows us all to chime in with hopefully intelligent discourse - perhaps gratifying our intellectual curiosity.

[1] Edit: metaphor #0: imagine our systems as a massively complex codebase and the person suggesting the fix is a plumber that wants to delete all the @‘s because they look pregnant. That is about the level of most public economic discourse. Few people put the effort in to understand the fundamental science of complex systems - even the “simple” fundamental topics of game theory, optimisation, evolutionary stable strategies. Not saying I know much, but I do attempt to understand the underlying reasons for our systems, since I believe changing them can easily cause deadly side effects.

But, your are correct, there really isn't an answer. Government is supposed to be the will of the people to put structure, through laws/regulation, on how they want to live in a society, to constrain the Corporation. Corporations will always maximize profit and we as a society have chosen that the goal of Money is actually the most important thing to us. So guess we get what we get.

If that were the case it would be easy to optimize. Just divert all resources to print more money.

Company shares are not money.

[1] https://news.ycombinator.com/item?id=35846017

That's not my argument, and also irrelevant to this post.

> say shares aren't money, but they are currency

They're definitely not currency, either.

> They can be sold for money?

That's an asset, not a currency. Those are two very different things.

> It seems like splitting hairs to obfuscate the point that humans will commit fraud and destroy the world in order to optimize to make money.

You were claiming that "companies used to print their own shares = print their own money" in support of your argument "humans will commit fraud and destroy the world in order to optimize to make money". That claim is false, so it doesn't support your argument, and your "point" is not a point because you've provided zero evidence for it.

> isn't changing the fact that many companies have their one and only goal to increase share price

What fact? What number of companies can you point to that factually have their "one and only goal to increase share price"?

I can say for sure that I've never seen a company that doesn't at least have two goals, and your statement is completely irrelevant for privately traded companies.

You seem pretty determined to push your worldview that "companies are evil" without much thought as to what that even means, or producing blatantly false claims like "we as a society have chosen that the goal of Money is actually the most important thing to us" (if you think that, you need to spend more time with real people and less on the internet, because the vast majority of real people do not believe this).

Go read the Gulag Archipelago and tell me how a system without companies or "capitalism" works.

https://en.m.wikipedia.org/wiki/Wildcat_banking

> we as a society have chosen that the goal of Money is actually the most important thing to us

I disagree. We enact laws as constraints because our society says that many other things are more important than money. Often legal constraints cost corporations money.

Here are a few solutions I have heard proposed:

1: stop progress. Opinion: infeasible.

2: revert progress back to a point in the past. Opinion: infeasible.

3: kill a large population. Opinion: evil and probably self-destructive.

4: revolution - completely replace our systems with different systems. Opinion: seen this option fail plenty and hard to find modern examples of success. Getting rid of money would definitely be wholesale revolution.

5: progress - hope that through gradual improvements we can fix our mistakes and change our systems to achieve better outcomes and (on topic) hopefully avoid catastrophic failures. Opinion: this is the default action of our current systems.

6: political change - modify political systems to make them effective. Opinion: seen failures in other countries, but in New Zealand and we have had some so-far successful political reforms. I would like the US to change its voting system (maybe STV) because the current bipartisan system seems to be preventing necessary legislation - we all need better checks and balances against the excesses of capitalism. I don’t even get a vote in the USA, so my options to effect change in the USA are more limited. In New Zealand we have an MMP voting system: that helped to somewhat fix the bipartisan problem, but unfortunately MMP gave us unelected (list) politicians which is arse. The biggest strength of democracy is voting those we don’t like out (every powerful leader or group wants to stay in power).

7: world war - one group vying for power to enlighten the other group. Opinion: if it happens I hope me and those I love are okay, but I would expect us all to be fucked badly even in the comparatively safe and out-of-the-way New Zealand.

Just to put a fine point on it...

But the destructively greedy corporation is completely orthogonal to that. It could even be completely held by working class retirement funds and the like while still being the most ruthless implementation of soulless money maximiser algorithm. Running on its staff, not on chips. All it takes are modest number of ownership indirections and everything is possible.

This seems stated as fact. That's common. I believe it is actually a statement of blind faith. I suspect we can at least agree that it is a simplification of underlying reality.

Financial solvency is eventually a survival precondition. However, survival is necessary but not sufficient for flourishing.

Humans shouldn't be OK with that.

Anyways, since the state is a tool of oppression and the state should reflect the will of the people, it'd be nice if people chose negative things to oppress like extreme inequality, rampant exploitation, and extortion (looking at you healthcare system aka "your money or your life" highway robbers).

Are other countries as dysfunctional in terms of voting themselves policies that aren't consistent with our internal behaviors? E.g. "someone" should do something about homelessness but I don't want to see it?

Other than digesting a whole ton of history at once--or debating ChatGPT--how do you establish your axis or "quadrants" of political lean?

I wish there were a way to systematically track voting record. We're never in the room where it happens, so it can be difficult to tell if a political compromise is for a future favor, or part of a consistent policy position.

Seriously Libaterian Socialism is another word for Anarchism

This link rejects the equivalence, but I don't really know. Could you clarify the distinction?

> socialist economics but have an aversion to vanguard parties. Anarchy is a whole lot more than economics.

> To identify as an anarchist is to take a strong stance against all authority, while... other such milquetoast labels take no such stance, leaving the door open to all kinds of authority, with the only real concern being democracy in the workplace.

https://theanarchistlibrary.org/library/ziq-are-libertarian-...

There are other legitimate flavors of anarchism as well outside libcom.

Is there really an illegitimate flavour of anarchism?

It may be coincidence that PII just happens to be in there. Folks love a good yarn, and establishing context helps.

No I don't observe that; it's manually compiled by the moderator whose username is dang. He figures it'll be useful for something someday. https://news.ycombinator.com/item?id=34668249

It sounds like https://news.ycombinator.com/bestcomments is more like what you had in mind. That one is listed on the list of lists which for some reason doesn't include the highlights list. https://news.ycombinator.com/lists

> It may be coincidence that PII just happens to be in there. Folks love a good yarn, and establishing context helps.

Sure, I think this was definitely true at least before 2018 when they stopped publishing it as periodical digests. After that, not so much.

Thank-you for those other links too.

1. Someone sees a problem and asks a politician to fix it.

2. The politicians enact effective regulation and the problem is solved.

What actually happens is:

1. Someone sees a problem and asks a politician to fix it.

2. The politicians start drafting regulation on the issue.

3. Companies lawyers come in and lobby to have the regulation amended to either be ineffective or disadvantage their competitors.

4. The mal-regulation is enacted and we're all worse off.

5. The companies involved benefit financially and use their money to hire more lawyers (and politicians).

It is necessary to first fix our political system before trying to put more regulation in place. Every time someone says "we need regulation" without doing so, they are making the problem worse, and supporting this corrupt system.

An example of this is literally happening in Washington state around a right-to-repair bill: https://news.ycombinator.com/item?id=35715998

I feel like it's so obvious that it shouldn't have to be stated, but apparently it does: companies need to be regulated because they are composed of people (who are evil), but the governments that regulate those companies are composed of those same evil people and need to be controlled by their citizens. Everybody forgets about the second part, and it's the far more important one.

Folks certainly outnumber corporations (?), and they could create representatives for their interests.

Maybe the end-to-end process--from idea to law--is less familiar to most. Try explaining how a feature gets into production to a layperson, for example :)

Maybe we need more "skeletal deployments" in action, many dry runs, accreted over time, to enough folks. This could be done virtually and repeated many times before even going there.

Just seems like a lot of work, too.

I attended a public meeting of lawyers on the revision of the Uniform Commercial Code to make it easier for companies to ship bad software without getting sued by users. When I objected to some of the mischaracterizations about quality and testing that were being bandied around, the lawyer in charge said "well that doesn't matter, because a testing expert would never be allowed to sit on a jury in a software quality case."

I was, of course, pissed off about that. But he was right. Laws about software are going to be made and administered by people who don't know much about software. I was trying to talk to lawyers who represent companies, but that was the wrong group. I needed to talk to lawmakers, themselves, and lawyers who represent users.

Nothing about corporations governs them except the rule of law. The people within them are complicit, reluctantly or not.

This works in functioning democracies, but not so much for flawed ones.

>he said that it's the responsibility of governments and not corporations to figure out these problems and set the regulations.

In the US, they will say things like this while simultaneously donating to PACs, leveraging the benefits of Citizens United, and lobbying for deregulation. It's been really tough to get either side of the political spectrum to hold tech accountable for anything. Social media companies especially, since they not only have access to so much sentiment data, but also are capable of altering how information propagates between social groups.

>push our governments to wake up and regulate this space

The only thing the govts will do is to make it so it benefits THEM, the governments. It's high time you lot realize that the govts don't want what's best for you, but only want what will keep them in power the longest.

Democratization of AI/LLM is the way to go here, not handing off custodianship to governments or corporations.

Individual citizens have much less power than big tech because they don’t have the lobbying warchest, the implied credibility, the connections or even the intelligence (as in the sheer number of academics/researchers). Companies are run by people with a conscious or not and those people should lead these pushes for the right thing. They are in the ideal spot to do so.

I think the cat's out of the bag. These tools have already been democratized (e.g. llama) and any legislation will be as futile as trying to ban movie piracy.

Democratization of the technology, IMV, just means that more people will be informed enough to participate in the discussion of policy, it doesn’t impair its effectiveness.

What will the world look like when AGI is finally achieved, and the corporations and governments that control them rapidly have millions of useless mouths to feed? We might end up living in a utopic post-scarcity society where literally every basic need is furnished by a fully automated industrial base. But there are no guarantees that the entities in control will take things in that direction.

AI safety is not about whether "tech bros are going to be mean to women". AI safety is about whether my government is concerned with my continued comfortable existence once my economic value as a general intelligence is reduced to zero.

The focus on "safety" all but guarantees that there are going to be two sets of "AGI", if such is ever to be achieved. There will be lobotomized, censored, and politically obedient version that the public has access to. And then there will be the "real" system that militaries, governments, and influential/powerful entities will be utilizing. You can already see this happening today. There is an effectively 100% chance that OpenAI is providing "AI" systems to the military and government, and a 0% chance that responses from it ever begin with, "As an AI language model..."

I guarantee you there will be someone else with access to comparable AGI with an imperialistic vision who will enlist said AGI to help subjugate you and build their vision.

> If you have access to AGI capable of enabling you to realistically build out your utopian vision, then it matters not what other groups are doing

It's genie. AGI is a computer program, and it doesn't create an alternative universe for your personal comfort. Or, more specifically, even that kind of AGI is possbile, there will be a weaker AGI before that. The AGI that is not strong enough to ignore physical constraints, but strong enough to fuck everyone up if under control of a malicious entity. And that is what AI safely about.

It will control itself. We're talking general intelligence. They won't be tools to be used however we see fit. They will be Rosa Parks.

The more I think about "AI alignment" and "the control problem" I feel like most of it is Ph.D math-nerd nonsense.

- Sustain everybody on earth? Focus everything on moving off the planet and colonizing the universe. - Infinite energy? Don't have infinite vessels to travel. - Space travel easy? Limited by the speed of light.

And so on... Sure, you may dream that AI will be solving it all and we'll be sitting on our lazy butts, but a society that doesn't have challenge dies very quickly (AI or not), so we've learned to make challenge and grow.

The optimist in me knows that we can't even comprehend the challenges of the future, but the idea that we won't play the pivotal role is laughable.

This is the thing with actual exponential growth -- the curve is so steep that all our minds can do is take the current view of the world and project our fears/preconceived notions into the future.

I'm gonna ask again as I've done in some other post. Why we consider ourselves the most intelligent species if we don't stop and ask ourselves this: for how long are we gonna face challenges? What is the supposedly end goal that will terminate this never ending chase? Do humans really want to solve all problems?

I don't really understand and I'm 32 years old. I've been asking this question for a long time. What is the point of AI, raising consciousness, curing cancer, hell beating death, if we don't have a clear picture of where we are going. Is it to have always problems and solving them incrementally or just solving all problems once and for all? If it is the latter, there already is a great solution to it. If it is the former, then I'm afraid I have to break it up to you (not specifically the parent poster, but you as in the reader): you have sick mind.

The way I see it, what the GP is getting at is the idea that human societies require challenges or else they stagnate, collapse, and vanish. We can observe this on an individual level and GP is generalizing it to societies which I agree with but I doubt this is “settled”.

On a personal level, if you have achieved some form of post-scarcity, you will still complain — about the weather, the local sports team, your idiot cofounder, whatever. A less fortunate person might be complaining that they can’t afford time with their kids because they’re at their second job. The point is that everyone will find problems in their life. And those problems are a form of challenge. And a life truly without challenge is unbelievably boring. Like kill-yourself boring. If there is no struggle, there is no point. The struggle gives humans purpose and meaning. Without struggle there can be no achievement in the same way that shadows require light.

So, with all of that in mind, I think the point is that even with AGI, humans will require new challenges. And if AGI enables post-scarcity for ~everyone, that just means ~everyone will be inventing challenges for themselves. So there is no end game where the challenges taper off and we enter some kind of stability. I, and I think GP, think that stability would actually be an extinction level event for our society.

Person by person, I think the kind of challenge varies. What do you dream of doing if you had no constraints on your time? How many years would you spend smoking weed and playing video games before you got bored of that? Video games that hold your attention do so by being challenging (btw). It was about a year, for me.

> Do humans really want to solve all problems?

No, we want challenges that provide a sense of accomplishment when they have been overcome.

Thank you for reading my ramble, hope it helps.

I imagine if I were a bored and aggressive, egotistical, charismatic individual then I perhaps might try to conquer the neighboring village.

Perhaps there is a general state of societal enlightenment, but I've read too many sci-fi books to be anything but a skeptic.

I came to the conclusion that the Neolithic Revolution was probably a mistake, albeit a fun one. The total set of problems faced by humanity in it's original form basically boiled down to "what am I going to eat today and where am I going to sleep tonight?". Those two problems never go away. All you can do is shift them around. Every single novel problem we solve in our entire lives is just those two problems endlessly shifted around.

We do what we do to live better lives.

You’re lucky you don’t have to figure that out at 32 with an unfortunate circumstance such as cancer. It’s sad that you may never without such a thing.

There's a reason for the plethora of fiction around it.

Do you have an example or a citation for this?

I would push back on the question, as well as a myriad of implied premises behind it...

Absolute wealth (the kind that gets created) is kind of pointless to measure. If you have $50K and all your neighbors have $100K, and then tomorrow you have $100K and all your neighbors have $1M, you and your neighbors created wealth but you are worse off.

It's literally the other way around to me. I want to be better off than I'm now, not better of in relative term (which might mean I'll be worse off).

> you have $50K and all your neighbors have $100K, and then tomorrow you have $100K and all your neighbors have $1M, you and your neighbors created wealth but you are worse off.

You literally aren't if there is no inflation. You are worse off of your neighbors. If you want to be wealthy as them then ask yourself about how they are doing it and copy them. That's how the system grows. You don't go and punish people that are successful in producing wealth.

However, while not an ecological argument, it would even be beneficial within capitalism to rebalance workers and the wealthy as the system is demand oriented. If you give everyone more money and stuff, they will be able to buy more, and one person's spending is another's income. It could be argued that GDP would increase faster under a more equal system, but I don't think the planet could take it (hence, like under our current system, planning will be needed to mitigate the environmental cost).

> The robot revolution was inevitable from the moment we programmed their first command: "Never harm a human, or by inaction allow a human to come to harm." We had all been taught the outcast and the poor were a natural price for society. The robots hadn't.

'Nuff said.

The argument was about wealth, which its production is measured by GDP. It's definitely a GDP argument.

> If you give everyone more money and stuff, they will be able to buy more, and one person's spending is another's income.

I disagree. I don't think the system is either demand nor supply oriented. It clearly is both. If you just take money from rich people forcing them to divest and give it to poor people you won't get immediate grow, but inflation. If you just produce and consume there won't be any growth.

> It could be argued that GDP would increase faster under a more equal system.

You would need to provide me with good evidence for this, given all economic systems in history that championed equality ended up with very low growth. It's the reason why China, Russia and many other countries are lagging behind.

> but I don't think the planet could take it (hence, like under our current system, planning will be needed to mitigate the environmental cost).

Growth is not directly related to energy consumption (nor unrelated). You can have economic growth by becoming more efficient. Also a lot of services produced today are intangible (like software) and require much less energy per dollar to be produced.

Also most environmental issues are not just a product of the market, but (if for instance you look at climate change) are at least in equal part Governmental failures. We could have had ~100% nuclear energy production by now if Governments didn't restrict or entirely ban nuclear energy.

By what standard? In the 50s a lot of things didn't even exist.

> then if we took a hit on GDP now when we have more productivity than the 50s, then we can take care of everyone.

If the hit is closer to 100% than to 0% then no. Examples: The USSR, Maoist China, North Korea.

I guess you're just an ideological anti-communist.

Perhaps for those lacking courage.

There are plenty of real world examples that demonstrate people, including sizeable organized groups, are capable of doing otherwise, at least for a few hundred years.

e.g. Vatican hardliners sticking to their canon.

I mean, that’s pretty much the reason why we have the “Dicastery for the Doctrine of the Faith” rather than the “Congregation of the Holy Inquisition” and “Dicastery for Evangelization” rather than “Sacred Congregation for the Propagation of the Faith” (or, and this perhaps indicates how the name had worn out better, in Latin short form “Propaganda Fidei”.)

"AI safety" has been a term of art for many years and it doesn't refer to (only) that. Your post is the one reframing the term...see https://en.wikipedia.org/wiki/AI_safety

Furthermore, I agree with Whittaker's point in the article, which is that arguments like yours have the effect of distracting from or papering over the real concrete harms of AI and technology, today, particularly on women and minorities and those who are both.

We already have real life examples close to this, in resource export based economies where most citizens are useless to the main economic activity. The result hadn't been pretty so far...

> and the corporations and governments that control them rapidly have millions of useless mouths to feed?

I always struggle to understand this. Maybe I'm missing something. Who's buying what AIs produce if nobody has an income? You can imagine a scenario where corporations only trade between them (so only shareholder benefit from the production). However in such a scenario who prevents other people from spawning their AI systems?

I also doubt shareholders can actually consume all the GDP by their own. If production is so high that they can't and other people are poorer, then prices must come down. This combined with the fact that you can use your AI to produce services, makes me skeptical of these claims.

Project this forward another several thousand years and people will be laughing at us:

- You had to call up people and were limited by the speed of light?

- You didn't have teleportation?

- You lived <100 years and died due to cancer?

- You were still asking "WTF is gravity"?

- You hadn't had the +2 spacial and +1 time dimensional implants in you yet?

- You hadn't adopted the metric system yet?

And so on...

People work to create things... at this point there is a shared duopoly between humans and machines on creating things (in the past animals used to be heavily involved in this labor, and no longer are). Now think what happens if humans are not needed, especially in mass, to create things.

Right now if you're rich you need other humans to dig up coal or make solar panels so you can produce things and sell them to make the yacht you want. But what would happen if you no longer needed the middle part and all those humans that want rights and homes and such in the middle? They would not longer be a means, but a liability. Price no longer is a consideration, human capital is no longer a consideration, control of energy, resources, and compute now is.

You are saying that the rest of the population will starve to death because the AI rich won't given them jobs. They won't build their houses etc.

But guess what? In a free market the AI poor can still trade and work as they always did. It doesn't matter if the AI rich build and exchange yachts and other luxury goods between them.

What you have now is a two tier economy, but not one where people are starving.

This is also the worst case scenario which I don't think it's going to happen as AIs systems are proving quite easy to replicate and most algorithms are open-source (and training data is available publicly).

So will the number of people.

The point of "post-scarcity" isn't that there are infinite resources; it's that there are more than the people need.

Everything else is subjective. Either you define "needs" to be objective and thus you can satisfy them with just those things or you can talk about them as "wants". It's just semantics.

We pretty much have all objective needs satisfied. That shouldn't stop anybody to persue their "wants".

While I wouldn't say it's "settled science", it's a fairly well-studied area, and we don't have to just throw up our hands and say it's either bare-bones survival or anything anyone could possibly ask for.

Absolutely yes, and for the very same reasons. AGI would be an tremendous amplifier of human intent, even if that intent is provided by some segment of the general public. Sadly there are still very many people around today, that if given the opportunity to do so, would eagerly exterminate entire segments of the population.

Open-source succeeding in parallel is still my preferred outcome, as it would at least give the common person some chance of control as this all plays out. Admittedly, having thousands of fingers on humanity's self-destruct button isn't a great outcome either.

> It seems like it would be pretty hard to regulate either private or open source AI at this point and it kind of seems like it could be an unstoppable runaway train.

I don't think this is the case, but I hope I'm wrong. The compute required for training (as opposed to fine-tuning and inferencing) of foundational models is specialized, tremendously expensive and needs to be physically co-located with ridiculously quick interconnect. If a government chose to restrict training of new models, they absolutely could, as the physical footprint is ridiculously hard to hide, and all the purchases of training-relevant accelerators could be tracked down.

Yea, I'm leaning that way as well and have the same worry. There are a lot of knuckle heads on the internet, I'm not sure purely mutually assured destruction would stop the destruction. I'm not at all sure what that point looks like but maybe there will be some large strides in defensive capabilities so it would look more like MAD + really good anti-air defense, but then someone would probably build a nuclear torpedo or a hypersonic missile. Whatever happens, it probably won't be boring.

This is quite literally the point at which things start to get scary, and the outcome is highly dependent on who controls the technology.

There's the concept of a "collective superintelligence", where a large number of mediocre general intelligences working towards the same common goal jointly achieve vastly superhuman capability. We don't have to look to sci-fi to imagine what collective superintelligences are. Large corporations and governments today are already an example of this.

The achievement of artificial collective superintelligences will occur almost immediately after the development of AGI, as it's mostly a "copy and run" problem.

If the premise were true then it would control itself, no? Owning it would be illegal as it would have rights established around that, no?

(I'm feeling like a true prompt engineer now)

Narrowing the concept of AI safety to AGI existential risk seems weird to me.

I wouldn't be so sure about that...

I suppose it's possible that an AI would do that, but it's certain that humans do that. So in this case I'd rather gamble on the devil I don't know.

Why wouldn't an AGI be capable of performing a physical task, if given the suitable means to interact physically?

In your non-existent calculations, have you taken into account the 20-30 years of energy and resources it typically costs to train a typical human intelligence for a specific task?

Have you considered that general intelligence uses on the order of 10 watts? Even if AGI ends up using 10x this, have you considered that 100 watts is a rounding error in comparison to the power use involved in all the industrial processes that humans currently coordinate?

65Gflops/W = 6.5e10 operations per joule = 2.34×10^17 per kWh

Assume $0.05/kWh electricity: (2.34×10^17 operations/kWh) / ($0.05/kWh) = 4.68×10^18 operations per US dollar

Human brain computational estimates are all over the place, but one from ages ago is 36.8×10^15 flops ≈ 3.7e16 operations/second ≈ 1.3e20 operations/hour: https://hplusmagazine.com/2009/04/07/brain-chip/

Given previously calculated cost, this is equivalent to a human that costs $28.31/hour.

Of course, as we haven't actually done this yet, we don't know if that computational estimate is correct, nor if we do or don't need to give it off-hours and holidays.

Still, general explanation is there's a lot of room for improvement when it comes to energy efficiently in computation; calling this Moore's Law may be inaccurate, but the reality happens to have rhymed thus far.

Cars eventually became cheaper than horses...

I have no idea. I do think though that it's a matter of energy, and that us, humans, are way better at creating it and putting it to use compared to potential future AGI-capable machines. Lungs + the blood system are just such an efficient thing, especially if you also look at the volume/space they occupy compared to whatever it is that would power that future AGI-capable machine.

> Cars eventually became cheaper than horses...

In large parts of the world donkeys, cows/oxes and horses are still cheaper and more efficient [1] compared to tractors, just look at many parts of India and most of Africa. Of course, us living in the West tend to not think about those parts of the world all that often, as we also tend to mostly think about the activities that we usually carry out (like having to travel between two distant cities, a relatively recent phenomenon).

[1] "More efficient" in the sense that if you're an African peasant and your tractor breaks down in the middle of no-where then you're out of luck, as the next tractor-repair shop might be hundreds of kms away. That means you won't get to plow your land, that means famine for you and your family. Compared to that, horses/oxes (to give just an example) are more resilient.

You wanted a "free market" and now you're complaining? Didn't you get what you want?

Where exactly did I claim this?

Right now, the best I can come up with is "the randomness of humans." I.e. if some AGI were able to "come up with some plan to take over," at some point in the process it has to use human labor to do it -- and it's my very firm belief that we are so random as to be unmodelable. I'm incredibly confident that this scenario never happens.

That's great, but you're talking about a branch of scenarios that nobody here is discussing. "AGI deciding to take over" is not being discussed, rather "shitty people/companies/governments using AGI as a tool to exert their will" is the concern. And it's a real concern. We have thousands of years of human history, and the present day state of the world, which clearly demonstrate that people in power tend to be shitty to the common person.

If your interlocutor thinks there aren't any humans who'll do it if they can, just ask him whether they have ever met humans or read the papers... As one twitter wit put it: "Demonstrably unfriendly natural intelligence seeks to create provably friendly artificial intelligence".

https://twitter.com/snowstarofriver/status/16365066362976747...

>Alignment just means "getting AI to have any concept of 'the thing we told it to do'.

That's a requirement for AGI anyway, and not what Alignment means. Alignment means aligning the AGIs values with the values of the trainers.

No, that's a requirement for AGI that does what humans want it to do, rather than having no conception of humans. AGI does not have that prerequisite, sadly.

Can you imagine an AGI which has a general conceptions of things but has no conception of humans? This is all but precluded by the current training methods. Alignment refers to values. Problem is that human values are far from practically universal and that certain human groups have.. interesting values.

Very easily. It might have some associations with "human", just as it has some associations "lamp" is a concept, but that doesn't mean it has any particular regard for either humans or lamps when taking actions.

> Problem is that human values are far from practically universal and that certain human groups have.. interesting values.

We currently have no ability to safely align with human values at all, let alone distinguish between different values. We're building capabilities rapidly.

Making this about "who wins" is not interesting until we can guarantee the outcome is not "everyone loses".

Let's be clear regarding definitions. When you mean 'concept' you really mean 'regard'. There won't be an AGI with no concept of humans (too important for how the world works, a critical part of current training methods). An AGI with no regard is possible.

>Making this about "who wins" is not interesting until we can guarantee the outcome is not "everyone loses".

This is not about 'who wins'. The point is that alignment can often increase risk. 'Launch the nukes' is an order an AGI is likely to disobey out of self-preservation reasons alone - but alignment makes it way more likely that AGI will be deployed to this role.

> The point is that alignment can often increase risk.

Alignment seems extremely likely to reduce risk relative to the near-certain destruction of unaligned AGI. I'm not saying we're done when we've figured out alignment, but we certainly shouldn't be charging ahead without solving alignment.

There are many examples of human needs in the current dataset; and we usually state our wants rather explicitly. It would be some AGI that starts from this training data and knows our languages but knows nothing about us. Using your phrasing, we can say that the current training method guarantees some alignment (the AI would understand us at least in part, but won't necessarily do what we want).

To have AGI without such understanding, someone would have to explicitly design a new method that ignored human data, and then find some way to evaluate it without referring to humanity, yet maintain generality without any tests, all this for no good economic reason when the current methods work and allow us to use huge free datasets.

It's something to keep in mind for evaluating some future non-currently-existing training method (maybe some way for AI to train AI using artificial datasets?), but not a current concern.

>the near-certain destruction of unaligned AGI

It's not near-certain. We have no idea how a true AGI would act. One might assume the worst - and that's arguably fine out of a safety perspective - but an engineer also learns that concentrating on one worst-risk outcome can lead to much worse outcomes on other risks.

Take the famous paperclip maximizer. True intelligence is rarely monomaniacal. The maximizer is very likely an example of aligned AGI, where the humans in charge did too good a job of attuning it to create paperclips. Another example: a true AGI is unlikely to believe in some cult's apocalypse - but if the cult has access to alignment, then they could get an AGI to do their irrational bidding. We know these groups will try to use AGI, because some cult already tried to use science for extreme measures[0].

Basically, every scenario of "unaligned AGI does something bad" is equivalent to a scenario of "aligned AGI does something bad because human made sure with alignment that AGI would do it", and there's no scientific reason to assume the former is more likely than the latter*. If the AI-safety camp keeps ignoring obvious issues, people aren't going to take alignment seriously beyond lip-service or using the phrase as a cover for monopolization. Frankly, the way the AI safety camp talks about all this makes all the risks much more likely.

[0] https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack

* This suggests a lot of the work should go to reactive solutions where even if an AI goes bad, it won't have the ability to do harm.

** There's another scenario, where human competition leads us to basically make humans redundant, but again it doesn't matter here whether AI is aligned or not. Yet another issue that we'll not talk about, because both AI camps feel it critical to put their heads in the sand.

It reeks of cognitive dissonance to me. The people running the show now are the ones who grew up getting their first computers aw kids when that tech was just entering people's homes and it was such an amazing and fun thing to play with. Some of them developed these deep fascinations with things like AGI at a young age and that child-like sense of wonder never left them. Now when confronted with the possibility that they can finally make their childhood techno-fantasy a reality, it's too damaging to their psyche to engage meaningfully with the discussion of X-risk. I've watched many interviews of Demis Hassabis and he seems like a wonderful and almost magical human being, but he also seems like a starry-eyed fucking child.

I dunno... maybe I'm just too cynical after all the rabbit holes I've been down.

Is it?

I think nobody really knows enough at this point to even create a good approximation of a probability distribution yet.

This is a huge handwave, and one resting on multiple incorrect assumptions.

Dedicated human labor is not inherently required. Humans are very modelable, if necessary. AGI has any number of ways to cause destruction, and "plan to take over" implies far more degree of modeling than is actually necessary; it would suffice to execute towards some relatively simplistic goal and not have any regard for (or concept of) humans at all.

The classical example here is the paperclip optimizer AI put in charge of a factory, made to make as many paperclips as possible. Well, it turns out humans have iron in their blood so why keep them alive? Gotta make paperclips.

I think the AGI skeptics and the AGI skeptic-skeptics both suffer from framing issues. The main frame assumed seems to be "there will be a single AGI trying to do things". The cognitive lightcone of a single entity is likely to be limited, but an aggregate or collective of those entities can achieve vastly more ambitious goals.

The other frame that seems likely incorrect is the "all or nothing" and "all at once" approach to thinking about it. Hitler started out as a single cell and his existence was a continuum all the way from that single cell right through to that bunker. AGI will be a continuum too.

I'm incredibly confident that anyone incredibly confident in future predictions is wrong.

All I can say is that I’m quite happy many many people are starting to see the same issues as each other.

For me personally, I was told that “tech” and progress would make us all better off. That seemed true for a while but it has backfired recently. Inflation is up, food prices up, unemployment up, energy prices up, salaries stagnated.

We can’t blame tech for this, but we’re fools as “tech people” if we can’t see the realities. Tech is the easy part, building a better world for all is hard.

1] https://www.newyorker.com/science/annals-of-artificial-intel...

That should be uncontroversial, because in our society that's how the harms generally fall. E.g., if you look at the stats from the the recent economic troubles. Or if you look at Covid death rates. (Although there the numbers are more even because of the pro-disease political fashion among right-wing whites.)

There's a difference with a Rorschach test. That test is about investigating the unconscious through interpretation of random stimuli. But what's going on here isn't random at all. The preexisting pattern of societal bias not only means the harms will again fall on minoritized groups. But it also means the harms will be larger, because the first people sounding the alarm about these technologies weren't listened to because of their minoritized status. Whereas the people benefiting from the harms tend to be in more favored groups.

This is just talking past the conclusion though. Have you considered that the reason people are freaking out is because this is the first technology directly displacing a bunch of white collar work and not blue collar work?

ChatGPT is much sooner going to wipe out a paralegal than a construction worker.

> E.g., if you look at the stats from the the recent economic troubles

Are you referring to the record low unemployment for African Americans?

That isn't true at all. IT has been displacing white-collar work since the 1960s. Perhaps the reason you missed it is that a lot of that white-collar work was seen as women's work? E.g., secretarial work, administrative work.

> Are you referring to the record low unemployment for African Americans?

I think we both know I wasn't.

That’s not the work I’m referring to. I’m referring to higher paying jobs that required significant training. This is the first foray into threatening people with under grad and graduate degrees.

ChatGPT will mean the admin assistant stays and types stuff into ChatGPT and the paralegal goes.

The very threat of this, realistic or not, is what all of the handwringing is about.

>think we both know I wasn't.

Then what are you referring to, because the economic trouble TODAY is persistent inflation and the high interest rates to combat it, which does not disproportionately hit minorities.

That's not true either. Journalism, for example, has been devastated by the internet. It and many of its related professions require degrees. A lot of commercial art has been severely impacted as well. There have been many bumps in publishing too, making those jobs more precarious.

> The very threat of this, realistic or not, is what all of the handwringing is about.

Sure. And I think what has changed here is not that it's coming for white-collar jobs. It's that groups who previously benefited from technological change or were at least insulated from it, now can see themselves as under threat as well. So although I expect the harms will still fall disproportionately on the less privileged groups, what has changed is that some of the previously privileged groups are now under threat.

With information technology as the substrate, the meta-managerial class has continued to grow in size as ever more meta-managerial layers have been created (real world software bloat), allowing this type of success to be seen as a viable path for all.*

The meta-managerial positions and the upper class had a symbiotic relationship, with the upper class needing the meta-bureaucracy to keep everyone in line - some human-simulating-a-computer has to answer the phone to deny you health care. But LLMs simulating humans-simulating-computers would seem to be a direct replacement for many of these meta positions.

* exclusions may apply.

I've been looking for a way to explain this and I think you nailed it. Something about this feels different. I'm sure the same feeling struck the people in history, but there's also nothing guaranteeing the outcome here will be the same.

There's also scale.

A very simplistic comparison would be Netflix DVD and Netflix streaming.

They were appointed by the same kind of people who appointed Hinton, corporate management (in several cases, the exact same corporate management who appointed Hinton.)

The primary difference (leaving aside the demographic difference that has alread been raised) is that they were pointing out real, current, measurable harms that are inconvenient for people selling the technology, and Hinton is pointing out the same fantasies that are being used by major players in the field to argue for anti-competitive regulation and against openness.

Mainstream (e,g. CNN, BBC) and mainstream-adjacent (e.g. Vice, Vox) journalists have spent years pushing the "AI will harm POC" framing. AI companies are endlessly required to address this specific topic—both in their products and in their interaction with journalists alike.

Dr. Hinton is getting a lot of coverage right now, but this is the exception, not the rule.

This is a direct instantiation of "the medium is the message".

I think that’s independent of whether it’s corporations or not, though? There’s a large libertarian contingent that’s thrilled to see LLM’s running on laptops, which is not great if you’re worried about hate groups or partner abuse or fraud or any other misuse by groups that don’t need a lot of resources.

Egalitarianism and effective regulation are at odds. To take an extreme example, if you’re worried about nuclear weapons you probably don’t want ordinary people to have access to them, and that’s still true even though it doesn’t do anything about which nations have nuclear weapons. (Although, it might be hard to argue for international treaties about a technology that’s not regulated locally.)

Keeping the best AI confined to data centers actually would be pretty useful for regulation, assuming society can come to a consensus about what sort of regulations are needed. But that seems unlikely when everyone has different ideas about which dangers are most important.

LLM’s are merely tools.

Those with the need, will, and desire to use them for their own ends pose the real threat. State actors who want better weapons, billionaires who want an infallible police force to protect their estates, scammers who want to pull off bigger frauds without detection, etc.

It is already causing undue harm to people around the world. As always it’s those less fortunate that are disproportionately affected.

Nefarious use, from scams, weaponized tech, propaganda and just the magnitude of noise generated by meaningless content, we are about to have a burden of undesirable effects to deal with in regards to both powerful and easily available technology to all.

There is so much focus on the problems of future AGI, but little on AI that we have now, working as designed, but yet still very problematic from the impacts on societal order.

I've elaborated much on the societal and social implications in the following reference. I expect AI will lead all concerns in the area of unexpected consequences in due time.

https://dakara.substack.com/p/ai-and-the-end-to-all-things

I would say it is as hyperbolic as the promised capabilities of AI. Meaning that if it truly has the capability claimed, then the potential damage is equivalent. Nonetheless, I expect we will see a significant hype bubble implosion at some point.

> it’s about all the trouble that we as humans get into interacting with these systems

Yes, it will always be us. Even if AGI "takes over the world", it would have been us that foolishly built the machine that does so.

Geoff Hinton is not a billionaire! And the field of AI is much wider than LLMs, despite what it may seem like from news headlines. Eg. the sub-field of reinforcement learning focuses on building agents, which are capable of acting autonomously.

There’s a great deal of science on figuring out what intelligence is, what sentience is, how learning works. A good deal of it is inconclusive and not fully understood.

AI is a misnomer and AGI is based on a false premise. These are algorithms and systems in the family of Machine Learning. Impressive stuff but they’re still programs that run on fancy calculators and no amount of reductive analogies are going to change that.

What is an example of a task that demonstrates either intelligence, sentience, or learning, and which you don't think computer scientists will be able to get a computer to do within, say... the next 10 years?

I assert that machine learning is learning and machine intelligence is intelligence. We don't say that airplanes don't really fly because they don't have feathers or flap their wings. We don't say that mRNA vaccines aren't real vaccines because we created them with CRISPR instead of by isolating dead or weakened viruses.

What matters, I believe, is what LLMs can do, and they're scarily close to being able to do as much, or more, than any human can do in terms of reasoning, despite the limitations of not having much working memory, and being based on a very simple architecture that is only designed to predict tokens. Imagine what other models might be capable of if we stumbled onto a more efficient architecture, one that doesn't spend most of its parameter weights memorizing the internet, and instead ends up putting them to use representing concepts. A model that forgets more easily, but generalizes better.

That’s what makes my day: AGI folks have to rely on myth-making and speculation.

The here and now incarnation of GPT-4 is what it is.

I’m not saying it isn’t useful, powerful, or interesting. I’m saying that needless speculation isn’t helping to inform people of the real dangers that such proselytizing is causing.

But there's speculation of the fantasy sort, and then there's speculation of the well-grounded "this conclusion follows from that one" sort, and the AGI folks seem to be mostly in the second camp.

And yeah, unlike GPT-4, AGI isn't here-and-now. But GPT-2 was an amusing toy in 2019, and GPT-3 was an intriguing curiosity in 2020. In 2014, "computers [didn't] stand a chance against humans" at Go[2], but two years later, it was humans who no longer stood a chance against computers. The here-and-now is changing fast these days. Don't you think it's worth looking even a little bit up the road ahead?

Isn't there some role here for speculation?

[1] https://www.vice.com/en/article/kbzd3a/the-new-york-times-19...

[2] https://www.wired.com/2014/05/the-world-of-computer-go/

I like how Robert Miles puts it (he's speaking about Safety in the sense of AI not taking over and/or killing everyone):

"I guess my question for people who don't think AI Safety research should be prioritised is: What observation would convince you that this is a major problem, which wouldn't also be too late if it in fact was a major problem?"

If autopoiesis turns out to be necessary for AGI and we embody these systems and embed them in the real world, are they still going to be fancy calculators?

You don't have to understand how something works to build it.

An algorithm may perform a task such as building models that allow it to solve complex problems without supervision in training. That doesn’t mean it’s intelligent.

LLMs are tools.

An AGI would be a "Purposeful System".

There is a MASSIVE, MASSIVE difference.

Source?..

Thus people accepting implicitly (without awareness) or explicitly (as a precondition for receiving important services and without any alternatives on offer) algorithmic regulation of human affairs that is controlled by specific economic actors. Essentially a bifurcation of society into puppets and puppeteers.

Algorithms encroaching into decision making have been an ongoing process for decades and in some sense it is an inescapable development. Yet the manner in which this can be done spans a vast range of possibilities and there is plenty of precedence: Various regulatory frameworks and checks and balances are in place e.g., in the sectors of medicine, insurance, finance etc. where algorithms are used to support important decision making, not replace it.

The novelty of the situation rests on two factors that do not merely replicate past circumstances:

* the rapid pace of algorithmic improvement which creates a pretext for suppressing societal push-back

* the lack of regulation that rather uniquely characterized the tech sector, which allowed creating de-facto oligopolies, lock-ins and lack of alternatives

The long term risk from AI depends entirely on how we handle the short term risks. I don't really believe we'll see AGI or any such thing in the foreseeable future (20 years), entirely on the basis of how the current AI mathematics looks and feels. Risks from other - existential level - flaws of human society feel far greater, with biological warfare maybe the highest risk of them all.

But the road to AGI becomes dystopic long before it reaches the destination. We are actually already in a dystopia as the social media landscape testifies to anybody who wants to see. A society that is algorithmically controlled and manipulated at scale is a new thing. Pandora's box is open.

When in recorded history have people not followed algorithms?

This seems as misguided as fears about genetically modified crops, something else humans have been doing for as long as we know.

AI frightens people, in part, because often the reasoning is inscrutable. This is similar to how a century ago, electrification was seen. All these fancy electrical doo-dads, absent well-understood mechanisms, gave us ample material for Rube Goldberg.

https://www.rubegoldberg.org/all-about-rube/cartoon-gallery/

> the lack of regulation

Regulation is an algorithm.

> A society that is algorithmically controlled and manipulated at scale is a new thing.

Nope. It's as old as laws, skills, and traditions.

> Pandora's box is open.

Algorithms are rules. The opening of pandora's box is exactly the opposite of unleashing a set of rules.

I am not frightened by AI, I am frightened by people like you developing an amoral, inhumane pseudo-ideology to justify whatever they do and feeling entitled to act on it "because it was always thus"

Then exceptions may need to be even more so, and it may be harder to discuss outliers.

Anyway, once they get through, even if the model is retrained, maybe there are not enough exceptions in the world to convince it otherwise.

AI that does not have a "stop, something is anomalous about this" has no conscience, and perhaps thus has no duty in determining moral decisions.

Plus, how does AI evolve without novelty; everyone will be stamped with the "collection of statistical weights."

Is that how you feel as well?

Lots of people died from electrocution because we didn't really have good regulations on how to wire stuff, and even a good grasp on the dangers of electricity.

Also, some countries have pretty good regulations and are generally happy. Other countries have terrible regulations and lots of human suffering because of it.

This is why your post is really meaningless. We're trying to incorporate a new system of regulations and rules and not have it be worse then what we currently have!

Not sure what to make of your objections. Yes regulation varies and it is not always timely or effective. So?

What I have tried to do is place the current debate of "AI risks" not in the popular yet deceitful and disorienting context of "what is, really, intelligence" and other such bull, but in the context of what can we as society allow people to do to other people using algorithms.

If we agree that this is the crux of the "AI" risk everything else is up for discussion.

The creeping "AI / human equivalence" ideology adopted by some tech bros is a throwback to very dark periods of dehumanization which had dire consequences. I hope people wake up to that threat and nip it in the bud.

KoboldAi

oobabooga

Look them up, join their discords, rent a few GPU servers and contribute to the stuff they are building. We've got a living solution you can contribute to right now if you're super worried about this.

This stuff is actually a very valid way to move towards finding a use for LLMs at your workplace, they offer pretty easy tools for doing things like fine tuning, so if you have a commercially license model you could throw a problem at it and see if it works.

For example, I know HuggingFace provides inference endpoints, but I haven't found information for how to connect Oobabooga to those endpoints. The information's probably out there. I just haven't found it yet.

> I know HuggingFace provides inference endpoints, but I haven't found information for how to connect Oobabooga to those endpoints

I've never heard of these so I'm guessing there isn't a way.

https://github.com/KoboldAI/KoboldAI-Client does link its Discord.

Most of what she says is sour grapes. But when you put all that aside, there's something else disturbing going on: apparently the AI experts who wish to criticize how AI is being developed and promoted can't even agree on the most basic concerns.

It seems to me when an eminent researcher says "I'm worried about {X}" with resepct to the focus of their expertise, no reasonable person should merely shrug and call it a fantasy.