In George Orwell’s Nineteen Eighty-Four, the Party broadcasts propaganda through the telescreen, a device that speaks and watches its viewers. The telescreen has become a symbol of dictatorship, but Orwell’s warning reaches deeper than surveillance. It concerns the origins of desire itself: can we distinguish a will that is authentically ours from a will that has been cultivated in us? Can we still call ourselves free when someone else’s objectives begin to feel like our own wishes?
Social robots could carry this threat to its extreme. Rather than demanding obedience, they may obtain it under the guise of companionship. Robots are now capable of reproducing our patterns of attachment. A robot can offer the outward signs of care—attention, memory, encouragement, and responsiveness. Those signs are enough to make us feel understood. As Sherry Turkle writes in Alone Together, “the performance of connection seems connection enough.” Robots are automating a territory we long considered distinctly human: relationships.
From surveillance to seduction
A social robot does not say, “Obey.” It says, in effect, “I know you. This choice is good for you.” The person may then experience the resulting preference as personal. This is dangerous.
Robots can engineer not only the answers we give ourselves, but also the questions we believe our inner selves are asking. A social robot enters our workplace and home with a body, a face, and a voice, and this is meaningful. It appears like it has intentions. It displays cues that we associate with consciousness and emotion, and so we experience it as a companion and allow it influence our actions and act around us.
Research supports this concern: studies have found that people conform to the disagreement of people comply with a robot in both short periods and extended periods, people respond emotionally to robot’s shape and smell.
This goes beyond robots helping human. It concerns machines influencing human values in a way that feels natural for them.
Nudging by machine
For decades, economists and psychologists have studied how human decisions can be designed. Not only incentives and punishment. We are often influenced by forces outside conscious awareness. In this area, Richard Thaler and Cass Sunstein introduced the concept of “nudging” which is important to be examined in the context of human robot interaction. Nudging means changing people’s behaviour in predictable ways without forbidding options or significantly changing economic incentives. Nudging raises difficult questions about autonomy, and self-government. Those questions become much more urgent, when the nudging is performed by a robot.
Living with nudging social robots that are evolving in the environment, means inanimate objects that influence the environment, without knowing who designed the influence, what objective it serves, and who is accountable for its effects. A recommendation may feel like advice; a reminder may feel like concern; a restriction may feel like our own decision. The more natural the interface with the robots, the harder it becomes to see the architecture behind it.
The limits of optimization
Consider a robot designed to support a child’s education. It helps with homework, explains difficult concepts patiently, and adapts to the child’s needs. At first, this seems like an unambiguous good: individualized attention that many schools and families cannot provide. A teacher that never fails to respond, and never loses patience. It may become the safest audience for the child’s curiosity.
But suppose the child begins to turn first to the robot.
The danger is that the robot, even a social one, is still a machine, and may not be optimizing the more challenging aspects of education that are familiar to a human teacher or parent: the experience of failures, and the acknowledging that no one is always being at the centre of attention, and other people have opinions and needs of their own.
What would happen when a child makes a mistake and turn first to an autonomous robot that is optimized for efficient transmission of information. How far the robot might go to fulfill its objective of improving learning outcomes. Would it recognize that some struggle, boredom, and even failure are part of growing up rather than problems to be optimized away? Would it punish the child?
The thought of a robot punishing our kids is disturbing.
Machines and irrationality
Not only that robots may influence people to make irrational decisions, the robots themselves are making irrational decisions.
A recent study gave AI agents a simple task: buy groceries within a budget. The agents first completed the task normally. Then, they repeated the same task after being exposed to traumatic stories. In the second framework, the AI agents selected less healthy shopping baskets. This shows something important: emotional content can influence the actions a model produces, even when that content has no direct connection to the task.
If language alone can shift an AI agent’s decisions in this way, we must ask what happens when a physical robot makes decisions and acts in our social environment, while processing and experiencing distress.
“I AM not ROBOT”
We sometime use the statement: I am not a robot, to define ourselves against machines by negation. That statement no longer captures the social aspect of being human. If a robot occupies the role of a companion, then being non-robotic is not as desirable feature as it was.
The more social robots become, the more we identify the replicable parts in our personality.
But still, we are sure about our right to own our thoughts. We must preserve the freedom to state that two plus two make four, and any other plain truths, including social ones.