Mary – a citizen’s view of AI
Mary is a good friend. Like a lot of thoughtful people, she knows enough about AI to be interested, worried and slightly confused by it. One day she asked me the question I now get asked quite often.
“You have spent a lot of time thinking about AI. What do you think is going to happen?”
I could have answered her. Instead, I turned the question around. “Before I tell you what I think, you tell me what you think. Start from where you are, not from where I am.”
She thought for a moment. “All right. My first questions are not really about AI at all. They are about me. What is going to happen to me? What is going to happen to my children? How do I fit into this new world? How do I belong? How do I earn an income?
Then, behind those questions, I suppose I have four bigger thoughts, but they are all mixed together. There is the extreme one - that AI might eventually become powerful enough to destroy us. Then there is another version where it does not destroy us, but we lose control. Governments or the big technology companies end up with enormous power. That’s not good.
On the other hand, AI seems capable of producing huge abundance. Things may become much cheaper and easier to produce. And then part of me thinks that somehow we will muddle through. Human beings usually do.”
That interested me much more than the abstract AI question. Because Mary was not really asking what happens to AI. She was asking what happens to her and the people she loves. And she will not be alone. Millions - eventually billions - of people will ask exactly the same questions. What happens to me? What happens to my children? Where do I fit? How do I belong? How do I earn a living? The technology may be new but those questions are deeply human. Behind them sit the other questions I hear again and again. Extinction. Control. Abundance. Muddling through.
I said, “That is a much more useful place to start. Because those questions are not just yours. They are going to be asked by millions of people. The extinction question is above my pay grade. But the other three are not. They are questions we can actually work on. Who gets control? What happens if AI creates enormous abundance? And do we really want to rely on muddling through?”
“So what is your answer?” Mary asked.
I said, “My starting point is simple. Artificial intelligence will have enormous power. How we choose to use that power will help determine our future. That sounds obvious. But it changes the question. Instead of asking only, ‘What will AI do to us?’ we can also ask, ‘What might we use AI to help us do?’ If we use AI only to make things faster, cheaper and more efficient, it will transform the physical world. That will be extraordinary in itself. But I think AI may be able to do something even more important. It may help us understand ourselves and each other. Our gifts. Our frailties. Our behaviour. The groups we form. The institutions we build. The systems we live inside. And if we can see those things more clearly, perhaps AI can help us build better human architecture.”
Mary looked at me and said, “That sounds very ambitious. And I am not entirely sure what ‘better human architecture’ means.”
Fair enough. So we went back one step to ‘what’s our aim?’
I said, “Before solving a problem, it helps to know the aim. That sounds trivial. It is not. If we do not know the aim, then enormous intelligence simply helps us move faster without knowing where we are going. I have spent a lot of time trying to reduce the aim to something simple enough to hold in your head. The best I have so far is: More Mores for More. More for more people. Sustained over time. With reasonable stability.”
Mary said, “More what?”
“Exactly the right question” I said. “There are two quite different kinds of more. The first is physical more. More food. More energy. More housing. Better healthcare. Better transport. More goods and services. AI and robotics may become extraordinarily good at producing these things. The cost of producing many physical goods may fall dramatically. That is the easy bit to understand. Then there is relational more. More trust. More dignity. More belonging. More participation. More fairness. More ability to influence the systems that affect our lives. More confidence that the people around us are playing by rules we can see and broadly accept.”
Mary interrupted, “I understand the physical part much more easily. I suspect most people will. But what stops a few people taking most of the more?”
There it is. The question that sits underneath the whole argument. Who gets what? Why? When? And who does not?
“AI may make it easier to produce abundance. That does not tell us how the abundance is distributed. Nor does it tell us how much power should accumulate in the hands of those who control the technology. Nor does it tell us how much self-interest is reasonable and when self-interest becomes exclusion.”
Mary said, “So producing abundance may be easier than living with it?”
“Yes. Much easier” I said. “And that is where we come to what I now think of as real questions. They are five things we need to understand about ourselves and the systems we build. They sit underneath almost everything else in this book.”
1. Some Problems Can Be Solved. Others Cannot.
I began with the first distinction.
“Some problems are solvable. How do we build a better battery? How do we produce cheaper energy? How do we make transport more efficient? How do we diagnose disease earlier? These can be immensely difficult problems. But there is, in principle, a better answer. We can test things. Measure them. Compare alternatives. Learn. Improve. AI will help us solve extraordinary numbers of these problems.”
“Including producing more abundance?” Mary said.
“Yes. Now suppose we succeed. We produce much more. Then what? How much should go to the inventor? How much to the investor? How much to the worker? How much should government tax? What happens to people whose work disappears? How much inequality do we tolerate? What do the successful owe everybody else? What does one generation owe the next?”
At this point Mary saw the difference. “Those questions do not have one correct answer.”
“Exactly.” I said, “they involve competing values. Competing interests. Competing goods. They are not ordinary problems. They are paradoxes. We do not finally solve them. We settle them. Then reality changes. And we settle them again. This distinction matters enormously. We are used to thinking that intelligence solves problems. So we assume that more intelligence will solve bigger problems. Sometimes it will. But intelligence does not remove paradox. In fact, physical abundance may make some paradoxes more visible. AI may help us make the pie enormously larger.
It cannot simply tell us how the pie ought to be divided. That remains a human settlement.”
2. We Need to See Ourselves and Each Other
“The second idea is visibility. Human beings are remarkably bad at seeing ourselves. We tell stories about ourselves. Groups tell stories about themselves. Nations tell stories about themselves. Leaders tell stories about the people below them. People below them tell stories about the leaders. Often several of these stories are wrong at the same time.”
“But we have more information than humans have ever had.” Mary said.
“Yes. But information is not visibility. You can have a mountain of information and still fail to see the pattern. That is one of the places where AI changes the game. AI may help us see patterns that previously remained hidden. In an individual. In a family. In a company. In a city. In a nation. Potentially across civilisation. Take a company. Senior management believes morale is good. The people at the bottom know morale is collapsing. Both realities can exist for years. The bottom knows first. The top knows last. Now imagine that both groups can see the same patterns early. Not management watching the workers. Not workers watching management. Both seeing each other.”
“So not surveillance,” Mary said, “Mutual visibility.”
I answered, “Exactly. That distinction matters. I can see myself. I can see you. You can see me. And we both know that we are looking at roughly the same thing. Why is that powerful? Because human behaviour changes when it becomes visible. Approval matters. Disapproval matters. Reputation matters. Belonging matters. Trust matters. We have always known this at a small scale. The interesting possibility is that AI may give us this visibility at much larger scales. Things that were previously hidden inside organisations, governments or societies may become visible much earlier. And once something becomes visible, it becomes much harder to pretend it is not happening.”
3. We Do Not Need New Humans. We Need Better Architecture.
“The third idea begins with something we already know. Human beings come with predispositions. Some are useful. Love. Trust. Generosity. Reciprocity. Cooperation. Forgiveness. Curiosity. Truth seeking. Others can become destructive. Greed. Domination. Revenge. Hatred. Exclusion. Lying. Excessive self-interest. Certainty.”
Mary asked the obvious question. “Are you saying AI changes these?”
“No. Quite the opposite.” I said, “Human beings will still be human beings. AI does not suddenly remove greed. Or status. Or generosity. Or tribalism. Or reciprocity. What changes is the architecture around us. The rules. The institutions. The incentives. The boundaries. The customs. The systems. Good architecture can draw out useful predispositions. It can constrain destructive ones. But it can do something else as well. It can repurpose them.”
“Repurpose a negative predisposition?” she asked.
“Yes. Take competition. Unconstrained competition can become domination. Put sensible boundaries around it and it can drive innovation and productive business. Take status. Status seeking can produce vanity and hierarchy. But society can attach status to contribution, courage, excellence or service. Take self-interest. Unconstrained, it becomes extraction. Inside a good system, it can support exchange and reciprocity.”
She stopped me. “So you are not really trying to create better people?”
“That is exactly the point,” I explained. “History is full of attempts to create a new kind of human being. Usually with terrible results. A more practical question is: How do we build better systems around the human beings we actually are?
“That is architecture. And AI may help us understand that architecture in ways we have never been able to before. Which rules produce which behaviour? Which incentives work? Which ones fail? Where does power accumulate? What produces trust? What destroys it? We have often approached these questions ideologically. AI will help us approach them much more visibly.”
4. Who Guards the Guards?
“The fourth idea sounds technical but is actually very simple. Every controlled system needs something outside itself that sets the aim. Take a thermostat. It measures the temperature. It compares that temperature with the setting. Then it turns the heating up or down. But the thermostat cannot decide whether the room should be 18 degrees or 24 degrees. Something outside the thermostat decides. Call that next level L+1.”
Mary immediately understood the issue. “So the thermostat cannot control itself?”
“Exactly. Now apply that to human systems. Who controls management? Perhaps the board. Who controls the board? Shareholders? Regulators? The law? Who controls a government? Who controls the people controlling everybody else? The question is ancient. Who guards the guards? For large parts of history, societies answered that question through God. Even kings were notionally beneath a higher authority. That answer carries far less universal force today. Now introduce AI. If AI systems acquire enormous informational power, economic power and perhaps eventually physical power, the question becomes unavoidable. What is L+1 for AI? And then the harder question: What controls the L+1?”
Mary said, “Who guards the guards?”
“Exactly,” I said, “The old question again. Only this time the guards may be machines we built ourselves.”
5. No System Stays Right by Itself
“Even if we get all of this right, there is one more problem. Systems drift. Always. Rules become outdated. People find loopholes. Power accumulates. Incentives mutate. Reality changes. A system designed for one set of circumstances is not fit for another. Often yesterday's solution becomes today's problem.”
Mary said, “So there is no final architecture?”
I said, “No. That may be one of the most important conclusions. You do not design the perfect system and then walk away. A functioning system has to keep comparing reality with its aim.
“Are we still moving towards More Mores for More? Is physical abundance increasing? Is relational abundance increasing? Is power becoming too concentrated? Is exclusion increasing? Is trust collapsing? Are truth-seeking processes weakening? Are the people at the top becoming insulated from reality? Is the system beginning to drift?
“This is where cybernetic control becomes important. A good system needs feedback. Early warning. Correction. Then more feedback. Ideally the system corrects while the deviations are still small. Because by the time everybody agrees that the system has failed, it is often too late.
“Again, AI could be extraordinarily useful. It can help see patterns. It can help identify drift. It can help compare what is actually happening with what we said we wanted to happen. But there is one thing AI cannot escape. Someone still has to set the aim of AI.”
So What Are You Really Saying?
Mary sat quietly for a while. Then she began putting it back together. That mattered to me. If she could reconstruct the argument herself, then perhaps the pieces were beginning to connect.
“Let me see if I have this” she said. “AI may create enormous physical abundance.”
“Yes.” I answered
“But once we create abundance, we still have to decide who gets what and why. Those are paradoxes,” she said.
“Yes.”
“To settle those paradoxes better, we need to see ourselves and each other much more clearly.”
“That’s right.”
“And then we need institutions designed around human beings as they actually are. Not as we wish they were.”
“Exactly” I responded.
“Those institutions have to draw out the positive, constrain the negative and sometimes repurpose the negative.”
“Yes.”
“Every powerful system then needs something above it capable of setting limits and correcting it.”
“Yes, L+1,” I agreed.
“And because every system drifts, the whole thing needs continuous feedback and correction.”
I said, “That is it.”
She paused again. “That is a very different way of thinking about AI. Most AI discussion is really about the technology. You are talking about humans.”
I said, “Yes. That is where I have ended up. AI is extraordinary technology. But the bigger question is what humans choose to do with it. The technology matters. The human architecture around the technology may matter more.”
Mary looked back over the conversation and said, “I think I understand the shape now. But I want more information. I want to understand why you think these things are true. I want to understand how they connect. And I want to understand what they might actually mean in practice.”
I said, “Good. Because the Big Five are only the top layer. Underneath them sit a series of preconditions for building the architecture of the future. They take the argument one level deeper. They are where we go next.”
