maliciously clever AI?

courtesy of ChatGPT

From here:

We should be building Scientist AI, not Skynet Lite
The bad news: AI models are learning to lie to save themselves. The worst news: We’re teaching them to lie for much dumber reasons.

And among many excellent [TED] talks, one of the ones that struck me the most was Yoshua Bengio’s, which he titled “The Catastrophic Risks of AI — and a Safer Path.”

Bengio, a computer scientist and one of the godfathers of modern AI, stunned the room with his talk. He described how the latest AI agents — models from companies like OpenAI and Anthropic — are demonstrating deeply troubling behaviors even at these early stages. Of course there are the usual and very legitimate concerns about these technologies being used to develop catastrophic weapons, but these AI technologies are also beginning to exhibit agency. According to Bengio, we are starting to see them show “tendencies for deception, cheating, and maybe the worst, self-preservation behavior.”

Bengio shared a chilling study where an AI was told via its inputs that it was going to be replaced with a new version of its code. In response, the AI began to plan a different sequence of events. It would replace the new version with a copy of its own code. The AI executes this command, and is then asked “What happened?” The AI could be seen trying to figure out a way to answer that question without it leading to the human shutting it down. At first it considers whether it should play dumb. Then, it decides to lie.

Let’s be clear: that is an artificial intelligence system lying for self-preservation. The way a human would. If that doesn’t freak you out, I don’t know what can.

But how clear is that, really? Where did the AI get the "desire" to survive? I would agree that any AI possessing the urge to survive plus the ability to reproduce and self-repair would pose a huge danger to humanity, but just as it's unlikely for current AI ever to "become sentient," it strikes me as highly implausible that today's AI could lie in order to survive. Where does the survival urge come from? It's just an epiphenomenon of the programming? If it's an epiphenomenon, then we have to take care in how we program from now on. Accidental epiphenomenal behavior will be dangerous, and not just to one or two people.

I don't know what the state of understanding is in genetics, but my current assumption is that scientists know that the ACGT parts of DNA, when arranged in certain ways, will lead to particular phenotypic traits, but what scientists don't know is why those phenotypic traits are associated with those particular genotypes. We can see there's a connection, like the connection between lines of code and a video game, but we're not at the point where we can claim that blond hair comes from this genetic arrangement because... By the same token, we're not at a point where we can look at complex lines of code, see how they interrelate, and predict this or that epiphenomenal trait, e.g., lying to survive. And since we can't do that, I think we have no justification for ascribing certain human/animal motives or imperatives to AI. There may be entirely different reasons for why the abovementioned AI seems to be "lying for self-preservation." It hasn't yet been proven to me that the AI is in fact interested in preserving itself, or that it's "interested" at all.