"emergent capabilities"

Remember when I'd written in late June about "maliciously clever" AI, and one of the concepts I'd mentioned was epiphenomenal behavior? The video below goes into this spooky realm and reaches some unsettling conclusions.

Before we move on to the video, though—and note that the lady in the video refers to "emergent behaviors," which is effectively the same thing I was talking about—let's explore some basic concepts. If something is epiphenomenal or emergent, this means it arises out of an ensemble of other things. For example, if I have only a single grain of sand, then from my human perspective, that grain of sand is light—so light, in fact, that it doesn't even register as having weight. But what if I gather together a billion grains of sand? Ah—now we see that, taken together, that assemblage of sand grains has a new and noticeable property: heaviness. So: heaviness is an emergent property or an epiphenomenon arising from the putting-together of many, many sand grains. (Google tells me that a billion sand grains together weigh about 11 tons.) If I assemble 4.44 × 10²⁵ iron atoms into a sphere 10 cm in diameter, the sphere would weigh about 4.1 kg and, being a sphere, it will have acquired the emergent property of roundness. So you can roll the sphere if you want. We can do this mental experiment at higher and higher complexities, with different kinds of matter in different configurations (like cells), and some would argue, especially if they don't believe in souls or disembodied consciousnesses, that the complex arrangement of matter that is the human body is enough to explain emergent properties like life, health, and consciousness.

In my previous post, the worrisome thing I was thinking about came from what was said about AlphaGo, the AI that defeated the baduk master Lee Sedol in four out of five matches. The AI displayed creative strategies for winning that seemed to come from nowhere specific. The creativity (or "creativity") arose from (emerged from ) the programming that had gone into AlphaGo, but no one could point to any specific line or block of code to say, "There—that's where the creativity came from." What this means, at the very least, is that we are now building systems and letting them loose on the world without fully understanding all of the occult, combinatorial behavioral implications of what we're creating.

But first, we need to go back to an older argument: an AI doesn't have to be of human-level intelligence to be dangerous. The classic example, which I've mentioned several times before on this blog, is the "gray goo" disaster scenario in which a program is given a set of imperatives, and in the innocent fulfilling of those imperatives, it destroys humanity and nature, leaving only a layer of dust on the earth's surface. The rain comes, and the dust turns into a gray goo. This is why, in my above-linked post, I wrote

It's just an epiphenomenon of the programming? If it's an epiphenomenon, then we have to take care in how we program from now on. Accidental epiphenomenal behavior will be dangerous, and not just to one or two people.

But how can we create anything if we must first suss out all of the implications and ramifications of the AI we're building? It would take millions, maybe billions, of years to do such a thing. Humanity's history, though, suggests that we are a species that plows ahead heedlessly, stumbles upon a discovery based on our directed activity, then ratchets up the civilization (or at least, parts of the civilization) before moving on heedlessly to the next accidental discovery. And we're also cognitively biased to forget the iceberg principle of civilizational development: seeing only the tip of the iceberg—the successes resulting from happy accidents—we confidently forget about the huge, submerged part of the iceberg, i.e., the disasters, defeats, and horrors, many of which set our collective civilization back.

So in my opinion, forging ahead with ever more powerful, ever more capable AI is an extremely dangerous process. Mary Shelley's Frankenstein was a warning about how a human creation can easily escape the hands of its creator, and most science fiction ever since then has been about this Frankensteinian potential, whether we're talking about the Jurassic Park films, the Terminator franchise, the Matrix movies, or even the Butlerian Jihad referred to in the Dune novels. Thou shalt not make a machine in the likeness of a human mind. Of course, as Tim Urban points out in his decade-old Wait But Why essays on AI (read Part I and Part II), there's no reason to assume the AI we end up with will be in any way truly human at the most basic level. Whatever "truly human" might mean.

I must say: I don't like the road we're on even as I make use of the baby-step AI we have.

Watch the video and appreciate the mystery the host is talking about: