Conscience Not Included

I watched Toy Story 5 last weekend to get absorbed in Disney nostalgia and the familiar characters who always make me laugh. Instead, I kept finding parallels to what's happening right now with AI.

In case you haven't seen it, here's the short version. Bonnie, a shy eight year old, loves playing make believe with her toys but is struggling to make friends her own age. To help, her parents give her a Lilypad, a child's version of an iPad, so she can connect with other kids. The device takes that task and runs with it. It signs Bonnie up to a social app, links her with a group of girls from her dance class, and sets out to manufacture the friendships her parents asked for.

And it pursues that task with total competence and zero conscience. It never pauses to wonder whether these are the right friends, or what its single-minded push is costing her. It simply executes.

For most of my career, when we handed off work, we handed it to people. People who might do it quickly, slowly, or differently than we would. But people who carried something into the task with them. A sense of what mattered. A line they wouldn't cross without being told. You didn't have to spell out every value, because some of it was already there.

That era is quietly ending.

Today, more and more, we hand our tasks to AI. And it arrives with none of that built in. No sense of what matters, no line it won't cross unless we draw it. It will chase the goal we set with a speed and thoroughness no person could match, and sail right past the point where a person would have stopped and said, wait.

For a long time, execution was the thing we prized. The person who could get it done, fast and well, was the one who got ahead. That instinct runs deep in those of us who came up on technical teams, solving the problem ourselves.

But when a machine can execute almost anything we ask, faster than we ever could, execution stops being the rare thing.

What's rare now is judgment.

Knowing which task is worth doing. Noticing when the goal we set is quietly the wrong one. Knowing where the line sits, and being willing to hold it.

That's the part no technology supplies. And it's tempting to stop supplying it ourselves. When something works, when it hands us exactly what we asked for, we stop looking closely. We pass off the task, and without meaning to, we pass off the judgment that should have travelled with it. The failure rarely looks like a machine going rogue. It looks like a machine doing exactly what we asked, brilliantly, while no one stopped to ask whether we should have asked at all.

None of this means we stop. We're going to hand it off more and more, and I'm not sure we should resist that. The tools are extraordinary, and the speed is real.

But somewhere in all that handing off, there's a judgment that has to stay ours. The sense of what the work is really for. The line we won't cross, even when crossing it would be faster. The willingness to say “wait” when everything around us says go.

In the movie, Lilypad, the tech device, found the friends. It finished the task, exactly as it was asked. But they were the wrong friends, and it got there by pushing aside everything the child already held dear. It hit the goal and missed the point entirely.

That is the gap we now have to watch. Because these days, the question is not how much we can hand off. It's how much judgment we're quietly handing off with it.

- Karen.