Here's how artificial intelligence might really see the worldS

I've heard it said that one of the important functions of science fiction is to comment on the implications of new technology and what it might do to society. But sometimes you run up against a science fictional technology that has really obviously staggering implications — and yet the work doesn't seem to acknowledge that.

For me, the example that first comes to mind is Star Trek's transporters. As long as the transporters work by ScienceMagic(TM) all is more or less well, but begin to explain or explore and you're suddenly faced with some weird problems involving identity. During the original series, one episode glanced at the can of transporter worms. But Next Generation had bigger problems. The replicators were based on transporter technology—that's the can opener, isn't it. If the replicator can take a load of random stuff and turn it into a steady supply of tea, earl grey, hot, what's to stop the transporter from turning out an endless stream of identical Rikers? And at that point, what would it mean, if anything, to be someone?*

It doesn't really take the replicator to make that question askable, though. Identity is kind of weird to begin with. It's just that most of the time we don't think much about it. Because obviously I am me, right? And my body is me, and your body is you. Anything not my body isn't me. Except when you look, you find it's not so simple. Your brain can be convinced a missing body part is still there. It treats tools you use frequently as part of your body. Some people are convinced that particular body parts aren't really theirs.

There are cases where a person's left hand insists on doing something that person insists they don't want and never willed it to do. And while, for instance, most of us feel that we're located in our bodies, some people have the persistent feeling that their selves are located somewhere outside their bodies. Or even that their self doesn't really exist. Honestly, reading too much of this sort of thing can give you the heebie-jeebies.**

Or it gives me the heebie-jeebies, anyway, and so of course it turned out I had to read up some on the topic for my novel Ancillary Justice. The narrator of Ancillary Justice is the warship Justice of Toren, an artificial intelligence with thousands of human bodies slaved to it, the ancillaries of the title. Justice of Toren's ancillaries are essentially parts of its body, its hands and feet and eyes and ears. And it became clear, as I pondered the idea of the story, that Justice of Toren really did need to be the narrator. I was going to have to portray, from the inside, a character that had multiple human bodies and brains making up part of its body.

I started off, in the very beginning, with a snap-together ScienceMagic(TM) kit.*** But that wasn't very satisfying. I knew that at some point I was going to have to just say, "Yeah, go with me here, this just works this way," but I wanted to have some idea—even a vague one—of what kind of real world logic might lurk behind ancillaries, and hence the events of Ancillary Justice—events that happen because Justice of Toren is the sort of being it is.

And there is, indeed, some real world logic to it. Take alien hand syndrome, where a person's hand does things they never actually willed it to, and won't stop even though they really wish it would. Sometimes turns up in stroke patients, but it's most common in people who have undergone a corpus callosotomy, where the corpus callosum, the main connection between the right and left hemispheres of the brain, is cut. This is a surgery that's usually a last resort for very severe epilepsy. Now, this doesn't cut off all communication between the hemispheres—there are still some other (very limited) connections, and of course each hemisphere has an ear and an eye, and they generally both hear and see the same things.

What's really interesting is when they don't. Show a picture to only one eye—the left eye, say. Then ask a question—"what do you see?" They can't answer. The right hemisphere (which controls the left side of the body) knows what it saw, but it can't say, because it's the left side of the brain that handles speech. (For most of us, anyway. For some people it's the opposite side.) And the left half of their brain doesn't actually know what the right half saw.

Put headphones on our patient, and ask, into their left ear, for them to pick something up that's related to the image they're seeing, and that they can do—though only with the left hand. And here's where it gets kind of weird. Let's say you've shown our person's left eye a picture of a teacup, and asked their left ear to choose one of a variety of objects from the table in front of them, that might be connected with the image they're seeing. Their left hand picks up a teaspoon.

Now ask the patient—not just their left side—why they just picked up that spoon. Remember, only the left side of the brain can talk, and only the right side of the brain knows about the teacup and the request to pick something up. So if the left side answers, that answer is either going to be "I don't know," or it's going to be something like, "I just wanted to look closer at the pattern on the handle." It really has no idea why the left hand has picked up the spoon, but it integrates the action into its narrative of itself, it confabulates a motive.

The fact that one half of your brain could act without the knowledge or understanding of the other half is kind of amazing. But I find that moment of confabulation even more amazing. It speaks to the power of narrative, and how important narrative is in forming and maintaining identity that, when faced with some inexplicable action we've just taken, often our first reaction is to make up some reason for it. And once we've made up that reason, we're quite convinced by it. Yes, that must be the reason we picked up the spoon! It is the reason.

Here's what fascinates me about this: It's not like split-brain patients are aliens, whose minds don't work like ours. Not at all. Except for that surgery (which hopefully has saved their lives) their brains are like everyone else's. And of course it's not only split brain patients under special conditions who confabulate like this—it's actually quite common once you start looking for it. So, how many times have any of us done something that we're quite sure we know the reason for, but actually don't? Possibly quite a lot.

How many "I"s are there, getting input, thinking about that input, and acting—that we don't know about because we're papering it over with our confabulations? What grounds do we have for really believing we're any of us just one, unified person?

And if we're really assemblages of people—it's a big if, I know, but I'm all about the spec fic so those come kind of naturally—and if it's communication that makes the difference (the conditions that make split brain patients look kind of like they're two people in one body involve seriously restricting communication and information sharing between the two hemispheres of the brain, after all), and if really your body isn't a perfect boundary for your self, then what would you get if you could link together different brains in different bodies? Would real, complete telepathy make it hard to know where "I" left off and "you" began? How would you draw those boundaries?

Justice of Toren's self has thousands of bodies, each one with its own functioning brain—though for the purpose of the novel, I'm mostly only concerned with one twenty-body unit of ancillaries, and ultimately one, single body. How could such a being have (let alone maintain) a single sense of self? How does an entity like Justice of Toren handle a situation where part of it is doing, thinking, feeling things it doesn't want? How alien, to have a mind like that!

Or maybe not so alien. It's entirely possible we ourselves have minds a bit like that—just on a smaller scale, with fewer bodies to manage. We just don't normally look at ourselves that way, because it gives us the shivers and we've got a handy, running narrative of ourselves to make it all seem manageable and explainable.

Don't let anybody tell you stories aren't important. It's entirely possible that stories are what hold us together, as individuals. That we are the stories we tell about ourselves.

Ann Leckie is the author of Ancillary Justice (Orbit Books).

____

*It can be even more frustrating for a show like ST to look, every now and then, as though they're going to consider this question, only to fail each time to take it as far as it logically leads. On the other hand, sometimes you find someone has taken it to one of its logical conclusions and you maybe wish they hadn't.

**Which is almost certainly why Star Trek writers pretend none of it exists, or if they do get too close they throw a couple of gummy worms on the set and hope you won't look past those.

***ScienceMagic(TM) is of course owned by the genre's favorite purveyor of toys, Sufficiently Advanced Technologies.