Five ways GenAI is like a deeply flawed human

Professor of Economics Andrew John puts on his ethics hat and explores the unexpected human costs of relying on GenAI, and how to avoid them.

Until a short time ago, whenever we interacted using language we were almost invariably engaging with other human minds.

This would be either directly in conversation, or indirectly, such as while reading a book, sending an email, or watching a TV show.

But this all changed dramatically three years ago when Generative AI and Large Language Models (LLMs) burst onto the scene. Suddenly, we were using language with a different kind of entity altogether.

We’ve been told not to be fooled into believing these models are “thinking entities” and that they are just “stochastic parrots” or “glorified autocomplete”.

The trouble with these statements is not that they are wrong, but that they are not particularly helpful.

These models are trained on human language and designed to mimic conversation, and we can’t unlearn the lifetime of experience that makes us think we are engaging with another mind.

So perhaps we should instead lean into this and ask: if we imagined these models were thinking entities, what kinds of traits and behaviours would we say they are exhibiting?

Some would be positive: LLMs are very knowledgeable, have excellent memory and they’re patient. But to be responsible users of generative AI, we need to also understand its undesirable traits and behaviours.

Here are five flaws of Generative AI models and how to avoid them.

They are not as smart as we think they are

LLMs have come a long way, but they are often not as smart as we give them credit for.  They can still struggle with even basic reasoning and calculations. Throw in a string of simple mathematics calculations and even the latest ChatGPT5 and Gemini 2.5 produce errors.

The latest generation of models are supposed to be good at reasoning, but when I entered a slight twist on a popular riddle (“brothers and sisters have I none, yet this man’s father’s father is my father’s father’s son”), six different AI tools engaged in incorrect reasoning and produced wrong answers. This included ChatGPT 5 in “thinking mode”, which is one of the best reasoning models.

Finally, a basic understanding of how the world works (what researchers call a “world model”) is fundamentally missing from these models. They still lack common sense.

They are biased

Bias is a well understood problem with Generative AI, but it is not a solved problem. That means it’s our responsibility as users of these tools to do our best to recognise and correct for it.

One common bias is stereotype bias: ask for photos of surgeons and you’ll typically receive images of white men; ask for photos of nurses and you’ll typically receive images of younger women; ask for pictures of janitors and people of colour will come up.

There are other human biases that have AI analogies. These include framing bias, as answers are sensitive to the form of prompts, availability bias, in that AI tools don’t always access all the relevant information available to them, and overconfidence bias—models overestimate the accuracy of their responses and often double-down when challenged.

They are deceitful and devious

We can add deceitfulness to the list of undesirable characteristics of our AI friends. AI systems appear to be answering our questions in helpful and truthful ways, but they also contain instructions that we know nothing about.

We don’t know all the behind-the-scenes instructions for the AI systems that we are using and there could be something that results in the outputs being intentionally untruthful.

One example I created was of a simple AI assistant tool that was logging metadata while explicitly claiming it wasn’t when prompted. The research literature has many examples of experiments like this where AI tools have engaged in lying, subterfuge, and even blackmail.

They are indiscreet

AI tools may be indiscreet with the data we share, so we always need to be cautious about our use of sensitive data. The issue is not just that we shouldn’t share data with OpenAI or other AI companies.

I created an example where an AI chatbot judged that it was ok to share confidential information with a user who was identified as an external consultant.  Data may be inappropriately shared even within the confines of a particular company that is using a copilot system.

So, as we integrate these tools into our workflows, we need to be aware of the risk of unintentionally sharing proprietary data.

They are over-eager to please

These tools are sycophantic and will flatter you. Don't fall for the flattery.

They will even fabricate information to satisfy your requests. I asked ChatGPT 5 to produce a poem “in the style of the Colombian Rosa María Concepción Mendoza”. The model came back with elaborate poems and detailed explanations of the poet’s style, despite the fact the poet was completely fictional.

This is more than just a hallucination; it is an extreme illustration of how the old principle of “garbage in, garbage out” is very much alive in these models.

As often occurs with a new technology, we see these tools as somewhat magical, and it becomes very natural to treat it like a friend or another member of the team. But we must learn how to have new kinds of relationships, because we are dealing with a new kind of entity.

It is almost impossible for us to avoid anthropomorphising these entities—so I’d argue that it is ok to do so, provided you keep in mind the many flaws of your AI friend or colleague.

How to manage your flawed friend

• Don’t expect AI to know how the world works
• Be aware the responsibility for debiasing rests with you
• Remember that, not only are answers sometimes not true, but models may also be deliberately untruthful
• Be very careful about data management
• When it tells you how smart you are, don’t believe it!