When AI Sounds Human: What should we teach young children about chatbots?

John Wright · 3 October 2026

As conversational AI becomes increasingly human-like, primary schools may need to teach children not only how to use AI, but what it is, and what it is not.

One of the most important things we may need to teach children about artificial intelligence is also one of the simplest:

It isn't a person.

That sounds obvious.

For adults who understand roughly how a large language model works, perhaps it is.

But today's AI systems are increasingly designed to make the distinction less obvious. They converse naturally. They use the first person. They respond sympathetically. They remember elements of conversations. They can speak in remarkably convincing voices.

Ask some systems a difficult question and they will tell you that they are “thinking”.

For a young child, what exactly are we expecting them to make of that?

As schools begin developing AI literacy, we understandably tend to concentrate on practical issues: hallucinations, checking information, personal data, copyright and appropriate use.

There may be something more fundamental that needs to come first.

Children need to understand what they are talking to.

The problem of a machine that behaves like a person

I've been thinking about this particularly after reading Beijing's latest guidance for AI in education.

One of its more striking features is its differentiated approach to younger children. Direct AI interaction is heavily restricted in the early years and the guidance specifically addresses anthropomorphic AI companions.

We don't need to adopt Beijing's age boundaries or its policies wholesale. Indeed, I'm not yet convinced exactly where our own age boundaries should sit.

But the underlying concern deserves serious attention.

Young children inhabit imaginative worlds very easily.

We actively encourage that. Storytelling, role play, imaginary characters and creative play are important parts of childhood and education.

The difference is that those experiences normally exist within recognisable structures.

A child understands that a character in a story is part of a story. An adult guiding imaginative play can watch how the child responds. A teacher notices facial expressions, body language, hesitation, excitement or distress and adjusts what happens next.

A chatbot doesn't have that awareness.

It receives an input and generates an output.

Yet the output may increasingly feel as though it comes from something that knows, thinks and understands.

That mismatch is making me wonder and is, in essence potentially concerning.

It sounds as though it understands you

Anthropomorphism is hardly new.

Human beings have always attributed human characteristics to animals, objects and machines. Children are particularly familiar with worlds in which animals speak and objects have personalities. Most modern cars and vans seem to have 'faces'.

Generative AI introduces something different.

The object talks back.

Not with a handful of pre-recorded responses, but through a potentially endless conversation tailored to what the user has just said. In the old days (not that I remember of course) when computer programs did this is was a simple algorithm and the user/child would quickly find the limits of what it could do and would understand how the interaction was essentially fake.

Talking back as a human might makes the illusion considerably more powerful.

An AI can tell a child that their idea is wonderful. It can express apparent concern. It can ask follow-up questions. It can produce language that sounds reassuring, interested or excited.

But there is an important difference between producing the language of understanding and actually understanding a child.

The AI does not notice the child's face (yet).

It does not know that the confident answer being typed is masking anxiety.

It does not see that a joke has frightened rather than amused them.

It cannot draw on years of knowing that individual child in the way a parent might, or on the professional understanding of a teacher who has watched that child interact with other people throughout the day.

The danger is not simply that AI occasionally produces incorrect information.

It is that a child may attribute capabilities and understanding to the system that simply aren't there.

Perhaps AI literacy needs to begin earlier than AI use

This creates an interesting challenge for primary schools.

I don't think the answer is simply to keep children away from artificial intelligence.

Increasingly, that won't be possible anyway. Even a Google search is no longer possible without AI help/interference.

Children will encounter conversational AI outside school. It will appear in devices, applications, toys and services used by families. The distinction between an “AI product” and an ordinary consumer product is likely to become increasingly blurred.

We therefore face something similar to the challenge schools have faced with online safety. This time though, it will be more pervasive.

We don't teach children about online risks because we want them using every online platform.

We teach them because those platforms exist in their world.

AI literacy should probably be approached in the same way.

For younger primary children, however, the starting point might not be teaching them how to write effective prompts.

It might be teaching them that a machine can produce language that sounds human without being human.

That is quite a sophisticated idea.

But it can be introduced simply.

The chatbot doesn't have feelings because it says, “I'm pleased”.

It isn't worried because it says, “I'm concerned”.

It doesn't know something merely because it answers confidently.

And when an AI interface says it is “thinking”, that doesn't mean it is thinking in the way their teacher, parent or friend thinks.

Perhaps some of the language used by AI companies doesn't help us here.

I would personally prefer interfaces aimed at children to use deliberately computational language, “processing”, for example, rather than language that encourages the impression of a human mind at work.

That distinction may become an important part of digital literacy.

This is also a design question

There is a danger of putting all the responsibility onto the child.

If technology companies deliberately create systems that look, sound and behave increasingly like people, we shouldn't then be surprised when children interpret them as people.

That makes anthropomorphism partly an education issue but also a product-design issue.

Schools and parents can teach children to recognise the difference.

But designers also make choices.

Does the system need a human name?

Does it need a human face?

Should it say “I'm thinking”?

Should it simulate emotional attachment?

Should an AI designed for a six-year-old behave as though it is their friend?

These aren't merely aesthetic choices.

They shape the mental model children form of the technology.

And that becomes particularly important when we move from an AI tool towards an AI companion.

There is an important difference between an AI that helps a child explore a science question and an AI designed to encourage an ongoing emotional relationship.

For primary-aged children, I think we should be extremely cautious about the latter.

A teacher relationship is different

This doesn't mean AI should never give a child positive feedback.

An educational system might reasonably say that an answer was thoughtful or encourage a pupil to try again. A carefully designed AI coach could potentially be very useful.

Indeed, one of the most promising uses of AI in education may be systems that don't immediately provide answers but instead question children, challenge their reasoning and encourage deeper thought.

But we shouldn't confuse that with the relational role of a teacher.

The teacher's job involves knowing the child, guiding their development and building a relationship over time.

That is fundamentally human work.

AI might operate within that educational relationship. It shouldn't become the relationship.

This is another reason why I would be considerably more comfortable with primary children using a school-controlled AI designed around a particular educational purpose than giving them unrestricted access to a general-purpose chatbot.

The question shouldn't simply be whether pupils have “access to AI”.

We should ask what kind of AI, designed for what purpose, behaving in what way and under whose oversight?

Schools won't control most children's AI experiences

There is a complication.

Some of the most significant AI interactions children have may not happen in school at all.

They will happen at home.

Over the coming years, increasingly capable conversational systems are likely to appear throughout ordinary family life. A child may encounter them through a smart speaker, tablet, television, toy or another device without anyone particularly thinking of the interaction as “using AI”.

This presents schools with a familiar problem.

When smartphones and tablets proliferated, society adopted them much faster than we developed a shared understanding of their implications for children.

Families made different choices.

Schools made different choices.

The evidence developed slowly while the technology spread quickly.

AI will follow a similar path, but potentially much faster.

The difficulty for schools is that the evidence surrounding children's interaction with conversational AI is still immature.

That makes it hard to tell parents exactly what they should or shouldn't do, and I'm not convinced that should be the school's role anyway.

Parents generally don't appreciate schools instructing them how to run their homes, particularly when the evidence itself is uncertain.

There is, however, an important difference between telling parents what to do and helping parents understand the questions.

Schools can explain why they are cautious about anthropomorphic AI.

We can explain the distinction between simulated empathy and human understanding.

We can explain why we would be concerned about a young child developing an unsupervised relationship with an AI companion.

And we can be transparent about what we know, what we don't know and why we have made particular decisions in school.

Parents can then make their own informed decisions.

Controlled experimentation rather than avoidance

Uncertainty shouldn't mean doing nothing.

Schools should be experimenting with AI.

But experimentation involving children needs to happen under carefully considered circumstances.

That means knowing what the system is designed to do.

It means understanding how children's data is handled.

It means considering the behaviour the system encourages.

It means ensuring appropriate adult oversight.

And, importantly, it means having a clear educational reason for putting the AI in front of the child in the first place.

The test shouldn't be:

“Would children enjoy this?”

Children would probably enjoy many highly anthropomorphic AI experiences.

The more useful question is:

“What does this interaction contribute to the child's learning or development, and are we comfortable with what the child may learn about AI from the interaction itself?”

Sometimes the answer will justify using it.

Sometimes it won't.

That is why controlled trials make more sense than either unrestricted access or blanket avoidance.

What should a six-year-old know about AI?

Perhaps this gives us a useful test for our emerging AI curricula.

By the end of Key Stage One, could a child begin to understand that:

an AI can sound like a person without being a person;
an AI can produce an answer without actually “knowing” something as a person does;
confident-sounding answers can be wrong;
an AI doesn't have feelings just because it uses emotional language;
they shouldn't share personal information with a chatbot unless a trusted adult has told them it is appropriate;
and they should speak to a real person when something is important, worrying or personal?

The language would obviously need adapting for age.

But those concepts may ultimately matter considerably more than whether a seven-year-old can construct the perfect prompt.

AI literacy cannot just mean becoming good at using AI.

It must also mean becoming good at understanding its limitations.

We need to teach the illusion as well as the technology

The extraordinary capability of modern language models is that they can produce convincing human language.

That is also one of their greatest educational challenges.

Adults already find themselves saying “please” to chatbots, thanking them, becoming frustrated with them and talking about what an AI “thinks” or “wants”.

We should therefore be realistic about the conceptual challenge we're presenting to children.

As conversational AI becomes more convincing, the distinction between a machine that performs the behaviours associated with understanding and a human being who actually understands becomes harder to see.

Schools have a role in making that distinction visible.

That doesn't require frightening children about AI.

Nor does it require pretending the technology has no value.

It means gradually giving children the knowledge they need to encounter these systems safely and intelligently.

Perhaps, before we teach young children what to ask artificial intelligence, we need to teach them 'what' is answering.

Get more like this

One practical idea, one resource, one piece of tech, weekly.