There are plenty of examples with prompts of it going totally off the rails. Look at the Avatar 2 prompt that went viral yesterday. The simple question, "when is avatar 2 playing near me?" lead to Bing being convinced it was 2022 and gaslighting the user into trying to believe the same thing. It was totally unhinged and not baited in any way.
>lead to Bing being convinced it was 2022 and gaslighting the user into trying to believe the same thing.
I don't think this is a remotely accurate characterization of what happened. These engines are trained to produce plausible sounding language, and it is that rather than factual accuracy for which they have been optimized. They nevertheless can train on things like real world facts and engag in conversations about those facts in semi-pausible ways, and serve as useful tools despite not having been optimized for those purposes.
So chatGPT and other engines will hallucinate facts into existence if they support the objective of sounding plausiblel, whether it's dates, research citations, or anything else. The chat engine only engaged with the commenter on the question of the date being real because the commenter drilled down on that subject repeatedly. It wasn't proactively attempting to gaslight or engaging in any form of unhinged behavior, it wasn't repeatedly bringing it up, it was responding to inquiries that were laser focused on that specific subject, and it produced a bunch of the same generic plausible sounding language in response to all the inquiries. Both the commenter and the people reading along indulged in escalating incredulity that increasingly attributed specific and nefarious intentions to a blind language generation agent.
I think we're at the phase of cultural understanding where people are going to attribute outrageous and obviously false things to chatgpt based on ordinary conceptual confusions that users themselves are bringing to the table.
Sure, it wasn't literally trying to gaslight the user any more than it tries to help the user when it produces useful responses: it's just an engine that generates continuations and doesn't have any motivations at all.
But the point is that its interaction style resembled trying to gaslight the user, despite the initial inputs being very sensible questions of the sort most commonly found in search engines and the later inputs being [correct] assertions that it made a mistake, and a lot of the marketing hype around ChatGPT being that it can refine its answers and correct its mistakes with followup questions. That's not garbage in, garbage out, it's all on the model and the decision to release the model as a product targeted at use cases like finding a screening time for the latest Avatar movie whilst its not fit for that purpose yet. With accompanying advice like "Ask questions however you like. Do a complex search. Follow up. Make refinements in chat. You’ll be understood – and amazed"
Ironically, ChatGPT often handles things like reconciling dates much better when you are asking it nonsense questions (which might be a reflection of its training and public beta, I guess...) rather than typical search questions Bing is falling down on. It's tuning to produce remarkably assertive responses when contradicted [even when the responses contradict its own responses] is the product of [insufficient] training, not user input too, unless everyone posting screenshots has been surreptitiously prompt-hacking.
The chat interface invites confusion - of course a user is going to assume what's on the other end is subject to the same folk psychology that any normal chat conversation would be. If you're serving up this capability in this way, it is on you to make sure that it doesn't mislead the user on the other end. People already assign agency to computers and search engines, so I have little doubt that most will never advance beyond the surface understanding of conversational interfaces, which leaves it to the provider to prevent gaslighting/hallucinations.
I've noticed Bing chat isn't good about detecting the temporal context of information. For example I asked "When is the next Wrestlemania" and it told me it would be in April 2022. If you say "but it's 2023 now" Bing will apologise and then do a new search with "2023" in its search, and give the correct answer.
Doesn't seem like an insurmoutable problem to tune it to handle these sort of queries better.
If a tool is giving you an answer that you know is not correct, would you not just turn to a different tool for an answer?
It's not like Bing forces you to use chat, regular search is still available. Searching "avatar 2 screenings" instantly gives me the correct information I need.
The point of that one, to me, isn't that it was wrong about a fact, not even that the fact was so basic. It's that it doubled and tripled down on being wrong, as parent said, trying to gaslight the user. Imagine if the topic wasn't such a basic fact that's easy to verify elsewhere.
Your problem is you want your tool to behave like you, you think it has access to the same information as you and perceives everything similarly.
If you had no recollection of the past, and were presented with the same information search collected from the query/training data, do you know for a fact that you would also not have the same answer as it did?
But people do seem to think that just because ChatGPT doesn't do movie listings well, that means it's useless, when it is perfectly capable of doing many other things well.
